Download PDF
Appl Clin Inform 2023; 14(04): 789-802
DOI: 10.1055/s-0043-1775565
Research Article

Engaging Multidisciplinary Clinical Users in the Design of an Artificial Intelligence–Powered Graphical User Interface for Intensive Care Unit Instability Decision Support

Stephanie Helman
1   Department of Acute and Tertiary Care Nursing, University of Pittsburgh, Pittsburgh, Pennsylvania, United States
,
Martha Ann Terry
2   Department of Behavioral and Community Health Sciences, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania, United States
,
Tiffany Pellathy
3   Veterans Administration Center for Health Equity Research and Promotion, Pittsburgh, Pennsylvania, United States
,
Marilyn Hravnak
1   Department of Acute and Tertiary Care Nursing, University of Pittsburgh, Pittsburgh, Pennsylvania, United States
,
Elisabeth George
4   Department of Nursing, University of Pittsburgh Medical Center, Presbyterian Hospital, Pittsburgh, Pennsylvania, United States
,
Salah Al-Zaiti
1   Department of Acute and Tertiary Care Nursing, University of Pittsburgh, Pittsburgh, Pennsylvania, United States
5   Department of Emergency Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, United States
6   Division of Cardiology at University of Pittsburgh, Pittsburgh, Pennsylvania, United States
,
Gilles Clermont
7   Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, United States
› Author Affiliations Funding U.S. Department of Health and Human Services; National Institutes of Health; National Institute of Nursing Research (5F31NR019725-02).
› Further Information
Also available at
    Permissions and Reprints

Abstract

Background Critical instability forecast and treatment can be optimized by artificial intelligence (AI)-enabled clinical decision support. It is important that the user-facing display of AI output facilitates clinical thinking and workflow for all disciplines involved in bedside care.

Objectives Our objective is to engage multidisciplinary users (physicians, nurse practitioners, physician assistants) in the development of a graphical user interface (GUI) to present an AI-derived risk score.

Methods Intensive care unit (ICU) clinicians participated in focus groups seeking input on instability risk forecast presented in a prototype GUI. Two stratified rounds (three focus groups [only nurses, only providers, then combined]) were moderated by a focus group methodologist. After round 1, GUI design changes were made and presented in round 2. Focus groups were recorded, transcribed, and deidentified transcripts independently coded by three researchers. Codes were coalesced into emerging themes.

Results Twenty-three ICU clinicians participated (11 nurses, 12 medical providers [3 mid-level and 9 physicians]). Six themes emerged: (1) analytics transparency, (2) graphical interpretability, (3) impact on practice, (4) value of trend synthesis of dynamic patient data, (5) decisional weight (weighing AI output during decision-making), and (6) display location (usability, concerns for patient/family GUI view). Nurses emphasized having GUI objective information to support communication and optimal GUI location. While providers emphasized need for recommendation interpretability and concern for impairing trainee critical thinking. All disciplines valued synthesized views of vital signs, interventions, and risk trends but were skeptical of placing decisional weight on AI output until proven trustworthy.

Conclusion Gaining input from all clinical users is important to consider when designing AI-derived GUIs. Results highlight that health care intelligent decisional support systems technologies need to be transparent on how they work, easy to read and interpret, cause little disruption to current workflow, as well as decisional support components need to be used as an adjunct to human decision-making.

Keywords

decision support systems - end user design - AI-derived GUIs

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by the Institutional Review Board.




Publication History

Received: 15 March 2023

Accepted: 26 July 2023

Article published online:
04 October 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Lim HC, Austin JA, van der Vegt AH. et al. Toward a learning health care system: a systematic review and evidence-based conceptual framework for implementation of clinical analytics in a digital hospital. Appl Clin Inform 2022; 13 (02) 339-354
    Article in Thieme ConnectPubMedGoogle Scholar
  • 2 Helman SM, Herrup EA, Christopher AB, Al-Zaiti SS. The role of machine learning applications in diagnosing and assessing critical and non-critical CHD: a scoping review. Cardiol Young 2021; 31 (11) 1770-1780
    CrossrefPubMedGoogle Scholar
  • 3 Sullivan C, Staib A, McNeil K, Rosengren D, Johnson I. Queensland digital health clinical charter: a clinical consensus statement on priorities for digital health in hospitals. Aust Health Rev 2020; 44 (05) 661-665
    CrossrefPubMedGoogle Scholar
  • 4 Patel VL, Shortliffe EH, Stefanelli M. et al. The coming of age of artificial intelligence in medicine. Artif Intell Med 2009; 46 (01) 5-17
    CrossrefPubMedGoogle Scholar
  • 5 Shortliffe EH, Sepúlveda MJ. Clinical decision support in the era of artificial intelligence. JAMA 2018; 320 (21) 2199-2200
    CrossrefPubMedGoogle Scholar
  • 6 Calzoni L, Clermont G, Cooper GF, Visweswaran S, Hochheiser H. Graphical presentations of clinical data in a learning electronic medical record. Appl Clin Inform 2020; 11 (04) 680-691
    Article in Thieme ConnectPubMedGoogle Scholar
  • 7 Cannesson M, Hofer I, Rinehart J. et al. Machine learning of physiological waveforms and electronic health record data to predict, diagnose and treat haemodynamic instability in surgical patients: protocol for a retrospective study. BMJ Open 2019; 9 (12) e031988
    CrossrefPubMedGoogle Scholar
  • 8 Helman S, Terry MA, Pellathy T. et al. Engaging clinicians early during the development of a graphical user display of an intelligent alerting system at the bedside. Int J Med Inform 2022; 159: 104643
    CrossrefPubMedGoogle Scholar
  • 9 Porter A, Dale J, Foster T, Logan P, Wells B, Snooks H. Implementation and use of computerised clinical decision support (CCDS) in emergency pre-hospital care: a qualitative study of paramedic views and experience using strong structuration theory. Implement Sci 2018; 13 (01) 91
    CrossrefPubMedGoogle Scholar
  • 10 Fareed N, Swoboda CM, Chen S, Potter E, Wu DTY, Sieck CJUS. U.S. COVID-19 state government public dashboards: an expert review. Appl Clin Inform 2021; 12 (02) 208-221
    Article in Thieme ConnectPubMedGoogle Scholar
  • 11 Matheny M, Israni ST, Ahmed M, Whicher D. Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril. NAM Special Publication Washington, DC: National Academy of Medicine; 2019: 154
    CrossrefGoogle Scholar
  • 12 Bersani K, Fuller TE, Garabedian P. et al. Use, perceived usability, and barriers to implementation of a patient safety dashboard integrated within a vendor EHR. Appl Clin Inform 2020; 11 (01) 34-45
    Article in Thieme ConnectPubMedGoogle Scholar
  • 13 Merkel MJ, Edwards R, Ness J. et al. Statewide real-time tracking of beds and ventilators during coronavirus disease 2019 and beyond. Crit Care Explor 2020; 2 (06) e0142
    CrossrefPubMedGoogle Scholar
  • 14 Chen L, Ogundele O, Clermont G, Hravnak M, Pinsky MR, Dubrawski AW. Dynamic and personalized risk forecast in step-down units. implications for monitoring paradigms. Ann Am Thorac Soc 2017; 14 (03) 384-391
    CrossrefPubMedGoogle Scholar
  • 15 Yoon JH, Mu L, Chen L. et al. Predicting tachycardia as a surrogate for instability in the intensive care unit. J Clin Monit Comput 2019; 33 (06) 973-985
    CrossrefPubMedGoogle Scholar
  • 16 Yoon JH, Jeanselme V, Dubrawski A, Hravnak M, Pinsky MR, Clermont G. Prediction of hypotension events with physiologic vital sign signatures in the intensive care unit. Crit Care 2020; 24 (01) 661
    CrossrefPubMedGoogle Scholar
  • 17 Barnett A, Winning M, Canaris S, Cleary M, Staib A, Sullivan C. Digital transformation of hospital quality and safety: real-time data for real-time action. Aust Health Rev 2019; 43 (06) 656-661
    CrossrefPubMedGoogle Scholar
  • 18 Limousin P, Azzabi R, Berge L, Dubois H, Truptil S, Gall LL. How to build dashboards for collecting and sharing relevant informations to the strategic level of crisis management: an industrial use case. 2019 International Conference on Information and Communication Technologies for Disaster Management (ICT-DM). 2019:1–8
    PubMed
  • 19 Dowding D, Randell R, Gardner P. et al. Dashboards for improving patient care: review of the literature. Int J Med Inform 2015; 84 (02) 87-100
    CrossrefPubMedGoogle Scholar
  • 20 Kyngäs H. Inductive Content Analysis. The Application of Content Analysis in Nursing Science Research. Springer; 2020: 13-21
    CrossrefGoogle Scholar
  • 21 Kurtzman G, Dine J, Epstein A. et al. Internal medicine resident engagement with a laboratory utilization dashboard: mixed methods study. J Hosp Med 2017; 12 (09) 743-746
    CrossrefPubMedGoogle Scholar
  • 22 Langkjaer CS, Bove DG, Nielsen PB, Iversen KK, Bestle MH, Bunkenborg G. Nurses' experiences and perceptions of two early warning score systems to identify patient deterioration-a focus group study. Nurs Open 2021; 8 (04) 1788-1796
    CrossrefPubMedGoogle Scholar
  • 23 McParland CR, Cooper MA, Johnston B. Differential diagnosis decision support systems in primary and out-of-hours care: a qualitative analysis of the needs of key stakeholders in Scotland. J Prim Care Community Health 2019; 10: 2150132719829315
    CrossrefPubMedGoogle Scholar
  • 24 Lomis KP, Jeffries A, Palatta M. , et al. Artificial Intelligence for Health Professions Educators. NAM Perspectives. Discussion Paper, National Academy of Medicine, Washington, DC; 2021
    PubMed
  • 25 Fletcher GS, Aaronson BA, White AA, Julka R. Effect of a real-time electronic dashboard on a rapid response system. J Med Syst 2017; 42 (01) 5
    CrossrefPubMedGoogle Scholar
  • 26 Schall Jr MC, Cullen L, Pennathur P, Chen H, Burrell K, Matthews G. Usability evaluation and implementation of a health information technology dashboard of evidence-based quality indicators. Comput Inform Nurs 2017; 35 (06) 281-288
    CrossrefPubMedGoogle Scholar
  • 27 Franklin A, Gantela S, Shifarraw S. et al. Dashboard visualizations: supporting real-time throughput decision-making. J Biomed Inform 2017; 71: 211-221
    CrossrefPubMedGoogle Scholar
  • 28 Matheny ME, Whicher D, Thadaney Israni S. Artificial intelligence in health care: a report from the National Academy of Medicine. JAMA 2020; 323 (06) 509-510
    CrossrefPubMedGoogle Scholar
  • 29 Mlaver E, Schnipper JL, Boxer RB. et al. User-centered collaborative design and development of an inpatient safety dashboard. Jt Comm J Qual Patient Saf 2017; 43 (12) 676-685
    PubMedGoogle Scholar
  • 30 Paulson SS, Dummett BA, Green J, Scruth E, Reyes V, Escobar GJ. What do we do after the pilot is done? Implementation of a hospital early warning system at scale. Jt Comm J Qual Patient Saf 2020; 46 (04) 207-216
    PubMedGoogle Scholar
  • 31 Wartman SA, Combs CD. Reimagining medical education in the age of AI. AMA J Ethics 2019; 21 (02) E146-E152
    CrossrefPubMedGoogle Scholar
  • 32 Strathdee SA, Hellyar M, Montesa C, Davidson JE. The power of family engagement in rounds: an exemplar with global outcomes. Crit Care Nurse 2019; 39 (05) 14-20
    CrossrefPubMedGoogle Scholar
  • 33 Goldfarb MJ, Bibas L, Bartlett V, Jones H, Khan N. Outcomes of patient-and family-centered care interventions in the ICU: a systematic review and meta-analysis. Crit Care Med 2017; 45 (10) 1751-1761
    CrossrefPubMedGoogle Scholar