Download PDF
Appl Clin Inform 2024; 15(02): 327-334
DOI: 10.1055/a-2272-6184
Research Article

Usability Testing of Situation Awareness Clinical Decision Support in the Intensive Care Unit

Matthew J. Molloy
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
2   Division of Hospital Medicine, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
3   Division of Biomedical Informatics, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Matthew Zackoff
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
4   Division of Critical Care, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Annika Gifford
5   Brigham Young University, Provo, Utah, United States
,
Philip Hagedorn
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
2   Division of Hospital Medicine, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
3   Division of Biomedical Informatics, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Ken Tegtmeyer
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
4   Division of Critical Care, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Maria T. Britto
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
6   James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
,
Maya Dewan
1   Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States
3   Division of Biomedical Informatics, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
4   Division of Critical Care, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
6   James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital, Cincinnati, Ohio, United States
› Author Affiliations Funding Dr. Dewan receives career development and research support from the Agency for Healthcare Research and Quality (K08-HS026975), which supported this study.
› Further Information
Also available at
    Permissions and Reprints

Abstract

Objective Our objective was to evaluate the usability of an automated clinical decision support (CDS) tool previously implemented in the pediatric intensive care unit (PICU) to promote shared situation awareness among the medical team to prevent serious safety events within children's hospitals.

Methods We conducted a mixed-methods usability evaluation of a CDS tool in a PICU at a large, urban, quaternary, free-standing children's hospital in the Midwest. Quantitative assessment was done using the system usability scale (SUS), while qualitative assessment involved think-aloud usability testing. The SUS was scored according to survey guidelines. For think-aloud testing, task times were calculated, and means and standard deviations were determined, stratified by role. Qualitative feedback from participants and moderator observations were summarized.

Results Fifty-one PICU staff members, including physicians, advanced practice providers, nurses, and respiratory therapists, completed the SUS, while ten participants underwent think-aloud usability testing. The overall median usability score was 87.5 (interquartile range: 80–95), with over 96% rating the tool's usability as “good” or “excellent.” Task completion times ranged from 2 to 92 seconds, with the quickest completion for reviewing high-risk criteria and the slowest for adding to high-risk criteria. Observations and participant responses from think-aloud testing highlighted positive aspects of learnability and clear display of complex information that is easily accessed, as well as opportunities for improvement in tool integration into clinical workflows.

Conclusion The PICU Warning Tool demonstrates good usability in the critical care setting. This study demonstrates the value of postimplementation usability testing in identifying opportunities for continued improvement of CDS tools.

Keywords

clinical decision support - usability testing - critical care

Ethical Approval

Our study was approved by the local Institutional Review Board (2020-0202) and was determined to be nonhuman subjects research.


Supplementary Material



Publication History

Received: 14 September 2023

Accepted: 18 February 2024

Accepted Manuscript online:
20 February 2024

Article published online:
01 May 2024

© 2024. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Burrus S, Hall M, Tooley E, Conrad K, Bettenhausen JL, Kemper C. Factors related to serious safety events in a children's hospital patient safety collaborative. Pediatrics 2021; 148 (03) e2020030346
    CrossrefPubMedSearch in Google Scholar
  • 2 Endsley MR. Measurement of situation awareness in dynamic systems. Hum Factors 1995; 37: 65-84
    CrossrefPubMedSearch in Google Scholar
  • 3 Endsley MR. Designing for situation awareness in complex systems. Proceedings of the 2nd International Workshop on Symbiosis of Humans, Artifacts and Environment: 2001
    PubMedSearch in Google Scholar
  • 4 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
    CrossrefPubMedSearch in Google Scholar
  • 5 Kushniruk A, Triola M, Stein B, Borycki E, Kannry J. The relationship of usability to medical error: an evaluation of errors associated with usability problems in the use of a handheld application for prescribing medications. Stud Health Technol Inform 2004; 107 (Pt 2): 1073-1076
    PubMedSearch in Google Scholar
  • 6 Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005; 74 (7-8): 519-526
    CrossrefPubMedSearch in Google Scholar
  • 7 Norman DA. The Design of Everyday Things. New York: Basic Books; 2013
    Search in Google Scholar
  • 8 Preece J, Rogers Y, Sharp H. et al. Human–computer interaction. New York: Addison-Wesley Publishing Company; 1994
    Search in Google Scholar
  • 9 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
    Thieme ConnectPubMedSearch in Google Scholar
  • 10 Akhloufi H, Verhaegh SJC, Jaspers MWM, Melles DC, van der Sijs H, Verbon A. A usability study to improve a clinical decision support system for the prescription of antibiotic drugs. PLoS One 2019; 14 (09) e0223073
    CrossrefPubMedSearch in Google Scholar
  • 11 Wright MC, Dunbar S, Macpherson BC. et al. Toward designing information display to support critical care. A qualitative contextual evaluation and visioning effort. Appl Clin Inform 2016; 7 (04) 912-929
    Thieme ConnectPubMedSearch in Google Scholar
  • 12 Bode L, Schamer S, Böhnke J. et al; ELISE Study Group. Tracing the progression of sepsis in critically ill children: clinical decision support for detection of hematologic dysfunction. Appl Clin Inform 2022; 13 (05) 1002-1014
    Thieme ConnectPubMedSearch in Google Scholar
  • 13 Ronan CE, Crable EL, Drainoni M-L, Walkey AJ. The impact of clinical decision support systems on provider behavior in the inpatient setting: a systematic review and meta-analysis. J Hosp Med 2022; 17 (05) 368-383
    CrossrefPubMedSearch in Google Scholar
  • 14 Dziorny AC, Heneghan JA, Bhat MA. et al; Pediatric Data Science and Analytics (PEDAL) Subgroup of the Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network. Clinical decision support in the PICU: implications for design and evaluation. Pediatr Crit Care Med 2022; 23 (08) e392-e396
    CrossrefPubMedSearch in Google Scholar
  • 15 Wright A, Phansalkar S, Bloomrosen M. et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform 2010; 1 (03) 331-345
    Thieme ConnectPubMedSearch in Google Scholar
  • 16 Shelov E, Muthu N, Wolfe H. et al. Design and implementation of a pediatric ICU acuity scoring tool as clinical decision support. Appl Clin Inform 2018; 9 (03) 576-587
    Thieme ConnectPubMedSearch in Google Scholar
  • 17 Dewan M, Muthu N, Shelov E. et al. Performance of a clinical decision support tool to identify PICU patients at high risk for clinical deterioration. Pediatr Crit Care Med 2020; 21 (02) 129-135
    CrossrefPubMedSearch in Google Scholar
  • 18 Dewan M, Soberano B, Sosa T. et al. Assessment of a situation awareness quality improvement intervention to reduce cardiac arrests in the PICU. Pediatr Crit Care Med 2022; 23 (01) 4-12
    CrossrefPubMedSearch in Google Scholar
  • 19 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
    CrossrefPubMedSearch in Google Scholar
  • 20 Brooke J. SUS: a retrospective. J Usability Stud 2013; 8 (02) 29-40
    PubMedSearch in Google Scholar
  • 21 Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Human Factors 2015; 2 (02) e14
    CrossrefPubMedSearch in Google Scholar
  • 22 Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. In: Proceedings of the INTERCHI'93 Conference on Human Factors in Computing Systems. Amsterdam, The Netherlands, The Netherlands: IOS Press 1993: 206-213
    PubMedSearch in Google Scholar
  • 23 Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003; 35 (03) 379-383
    CrossrefPubMedSearch in Google Scholar
  • 24 Brooke J. SUS—a quick and dirty usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B. eds. Usability Evaluation in Industry. England: Routeledge; 1996
    Search in Google Scholar
  • 25 Britto MT, Jimison HB, Munafo JK, Wissman J, Rogers ML, Hersh W. Usability testing finds problems for novice users of pediatric portals. J Am Med Inform Assoc 2009; 16 (05) 660-669
    CrossrefPubMedSearch in Google Scholar
  • 26 Nguyen TH, Cunha PP, Rowland AF, Orenstein E, Lee T, Kandaswamy S. User-centered design and evaluation of clinical decision support to improve early peanut introduction: formative study. JMIR Form Res 2023; 7: e47574
    CrossrefPubMedSearch in Google Scholar
  • 27 Kushniruk AW, Borycki EM, Kuwata S, Kannry J. Emerging approaches to usability evaluation of health information systems: towards in-situ analysis of complex healthcare systems and environments. Stud Health Technol Inform 2011; 169: 915-919
    PubMedSearch in Google Scholar
  • 28 Patel VL, Kannampallil TG. Cognitive informatics in biomedicine and healthcare. J Biomed Inform 2015; 53: 3-14
    CrossrefPubMedSearch in Google Scholar
  • 29 Dewan M, Vidrine R, Zackoff M. et al. Design, implementation, and validation of a pediatric ICU sepsis prediction tool as clinical decision support. Appl Clin Inform 2020; 11 (02) 218-225
    Thieme ConnectPubMedSearch in Google Scholar
  • 30 Martin B, Mulhern B, Majors M. et al. Improving pediatric intensive care unit discharge timeliness of infants with bronchiolitis using clinical decision support. Appl Clin Inform 2023; 14 (02) 392-399
    Thieme ConnectPubMedSearch in Google Scholar
  • 31 Wulff A, Montag S, Marschollek M, Jack T. Clinical decision-support systems for detection of systemic inflammatory response syndrome, sepsis, and septic shock in critically ill patients: a systematic review. Methods Inf Med 2019; 58 (S 02): e43-e57
    Thieme ConnectPubMedSearch in Google Scholar
  • 32 Chen JH, Asch SM. Machine learning and prediction in medicine—beyond the peak of inflated expectations. N Engl J Med 2017; 376 (26) 2507-2509
    CrossrefPubMedSearch in Google Scholar
  • 33 Park J, Zhong X, Dong Y, Barwise A, Pickering BW. Investigating the cognitive capacity constraints of an ICU care team using a systems engineering approach. BMC Anesthesiol 2022; 22 (01) 10
    CrossrefPubMedSearch in Google Scholar