Usability Testing of Situation Awareness Clinical Decision Support in the Intensive Care Unit

Abstract

Objective Our objective was to evaluate the usability of an automated clinical decision support (CDS) tool previously implemented in the pediatric intensive care unit (PICU) to promote shared situation awareness among the medical team to prevent serious safety events within children's hospitals.

Methods We conducted a mixed-methods usability evaluation of a CDS tool in a PICU at a large, urban, quaternary, free-standing children's hospital in the Midwest. Quantitative assessment was done using the system usability scale (SUS), while qualitative assessment involved think-aloud usability testing. The SUS was scored according to survey guidelines. For think-aloud testing, task times were calculated, and means and standard deviations were determined, stratified by role. Qualitative feedback from participants and moderator observations were summarized.

Results Fifty-one PICU staff members, including physicians, advanced practice providers, nurses, and respiratory therapists, completed the SUS, while ten participants underwent think-aloud usability testing. The overall median usability score was 87.5 (interquartile range: 80–95), with over 96% rating the tool's usability as “good” or “excellent.” Task completion times ranged from 2 to 92 seconds, with the quickest completion for reviewing high-risk criteria and the slowest for adding to high-risk criteria. Observations and participant responses from think-aloud testing highlighted positive aspects of learnability and clear display of complex information that is easily accessed, as well as opportunities for improvement in tool integration into clinical workflows.

Conclusion The PICU Warning Tool demonstrates good usability in the critical care setting. This study demonstrates the value of postimplementation usability testing in identifying opportunities for continued improvement of CDS tools.

Keywords

clinical decision support - usability testing - critical care

Ethical Approval

Our study was approved by the local Institutional Review Board (2020-0202) and was determined to be nonhuman subjects research.

Publication History

Received: 14 September 2023

Accepted: 18 February 2024

Accepted Manuscript online:
20 February 2024

Article published online:
01 May 2024

• References

• 1 Burrus S, Hall M, Tooley E, Conrad K, Bettenhausen JL, Kemper C. Factors related to serious safety events in a children's hospital patient safety collaborative. Pediatrics 2021; 148 (03) e2020030346
• 2 Endsley MR. Measurement of situation awareness in dynamic systems. Hum Factors 1995; 37: 65-84
• 3 Endsley MR. Designing for situation awareness in complex systems. Proceedings of the 2nd International Workshop on Symbiosis of Humans, Artifacts and Environment: 2001
• 4 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 5 Kushniruk A, Triola M, Stein B, Borycki E, Kannry J. The relationship of usability to medical error: an evaluation of errors associated with usability problems in the use of a handheld application for prescribing medications. Stud Health Technol Inform 2004; 107 (Pt 2): 1073-1076
• 6 Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005; 74 (7-8): 519-526
• 7 Norman DA. The Design of Everyday Things. New York: Basic Books; 2013
• 8 Preece J, Rogers Y, Sharp H. et al. Human–computer interaction. New York: Addison-Wesley Publishing Company; 1994
• 9 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
• 10 Akhloufi H, Verhaegh SJC, Jaspers MWM, Melles DC, van der Sijs H, Verbon A. A usability study to improve a clinical decision support system for the prescription of antibiotic drugs. PLoS One 2019; 14 (09) e0223073
• 11 Wright MC, Dunbar S, Macpherson BC. et al. Toward designing information display to support critical care. A qualitative contextual evaluation and visioning effort. Appl Clin Inform 2016; 7 (04) 912-929
• 12 Bode L, Schamer S, Böhnke J. et al; ELISE Study Group. Tracing the progression of sepsis in critically ill children: clinical decision support for detection of hematologic dysfunction. Appl Clin Inform 2022; 13 (05) 1002-1014
• 13 Ronan CE, Crable EL, Drainoni M-L, Walkey AJ. The impact of clinical decision support systems on provider behavior in the inpatient setting: a systematic review and meta-analysis. J Hosp Med 2022; 17 (05) 368-383
• 14 Dziorny AC, Heneghan JA, Bhat MA. et al; Pediatric Data Science and Analytics (PEDAL) Subgroup of the Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network. Clinical decision support in the PICU: implications for design and evaluation. Pediatr Crit Care Med 2022; 23 (08) e392-e396
• 15 Wright A, Phansalkar S, Bloomrosen M. et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform 2010; 1 (03) 331-345
• 16 Shelov E, Muthu N, Wolfe H. et al. Design and implementation of a pediatric ICU acuity scoring tool as clinical decision support. Appl Clin Inform 2018; 9 (03) 576-587
• 17 Dewan M, Muthu N, Shelov E. et al. Performance of a clinical decision support tool to identify PICU patients at high risk for clinical deterioration. Pediatr Crit Care Med 2020; 21 (02) 129-135
• 18 Dewan M, Soberano B, Sosa T. et al. Assessment of a situation awareness quality improvement intervention to reduce cardiac arrests in the PICU. Pediatr Crit Care Med 2022; 23 (01) 4-12
• 19 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
• 20 Brooke J. SUS: a retrospective. J Usability Stud 2013; 8 (02) 29-40
• 21 Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Human Factors 2015; 2 (02) e14
• 22 Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. In: Proceedings of the INTERCHI'93 Conference on Human Factors in Computing Systems. Amsterdam, The Netherlands, The Netherlands: IOS Press 1993: 206-213
• 23 Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003; 35 (03) 379-383
• 24 Brooke J. SUS—a quick and dirty usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B. eds. Usability Evaluation in Industry. England: Routeledge; 1996
• 25 Britto MT, Jimison HB, Munafo JK, Wissman J, Rogers ML, Hersh W. Usability testing finds problems for novice users of pediatric portals. J Am Med Inform Assoc 2009; 16 (05) 660-669
• 26 Nguyen TH, Cunha PP, Rowland AF, Orenstein E, Lee T, Kandaswamy S. User-centered design and evaluation of clinical decision support to improve early peanut introduction: formative study. JMIR Form Res 2023; 7: e47574
• 27 Kushniruk AW, Borycki EM, Kuwata S, Kannry J. Emerging approaches to usability evaluation of health information systems: towards in-situ analysis of complex healthcare systems and environments. Stud Health Technol Inform 2011; 169: 915-919
• 28 Patel VL, Kannampallil TG. Cognitive informatics in biomedicine and healthcare. J Biomed Inform 2015; 53: 3-14
• 29 Dewan M, Vidrine R, Zackoff M. et al. Design, implementation, and validation of a pediatric ICU sepsis prediction tool as clinical decision support. Appl Clin Inform 2020; 11 (02) 218-225
• 30 Martin B, Mulhern B, Majors M. et al. Improving pediatric intensive care unit discharge timeliness of infants with bronchiolitis using clinical decision support. Appl Clin Inform 2023; 14 (02) 392-399
• 31 Wulff A, Montag S, Marschollek M, Jack T. Clinical decision-support systems for detection of systemic inflammatory response syndrome, sepsis, and septic shock in critically ill patients: a systematic review. Methods Inf Med 2019; 58 (S 02): e43-e57
• 32 Chen JH, Asch SM. Machine learning and prediction in medicine—beyond the peak of inflated expectations. N Engl J Med 2017; 376 (26) 2507-2509
• 33 Park J, Zhong X, Dong Y, Barwise A, Pickering BW. Investigating the cognitive capacity constraints of an ICU care team using a systems engineering approach. BMC Anesthesiol 2022; 22 (01) 10