Avoiding Unintended Consequences of Pediatric Blood Order Set Updates through In Situ Usability Testing

Abstract

Background Blood product ordering is a complex process and mistakes can harm patients and lead to poor outcomes. Orders and order sets can be designed to help mitigate errors, but major changes in design can unintentionally cause new errors.

Objectives Our objective was as follows: (1) utilize formative in situ usability testing to iteratively improve the design of a redesigned blood product order set prior to go-live, (2) implement changes based on feedback derived from this testing, and (3) compare the error rate, system usability scale (SUS) score, time to task completion, and click counts between the prior order set in use at the time and the revised redesigned order set.

Methods A multidisciplinary project team convened to redesign blood product orders and order sets from scratch based on a review of the literature and benchmarking against four pediatric academic institutions with the goal of addressing prior ordering errors. The newly redesigned blood product order set was iteratively updated via in situ formative usability testing performed with available clinical users using a concurrent think-aloud protocol in real clinical environments. Errors, SUS scores, time to task completion, and click counts were assessed for the revised redesigned order set using summative testing.

Results Formative usability testing with 20 participants led to seven design changes in the redesigned order set which reduced the error rate at go-live. Summative usability testing showed that even though the usability scores were only slightly improved for the revised redesigned order set, the error rates in blood orders were significantly decreased.

Conclusion Usability testing can identify design errors early in the process which can be rectified prior to implementation, thus avoiding unintended consequences of changes.

Keywords

user-centered design - special transfusion - preventable safety - usability testing - VSA

Protection of Human and Animal Subjects

This work was felt to be primarily focused on quality improvement and therefore deemed nonhuman subjects research by the Institutional Review Board of Children's Healthcare of Atlanta.

Publication History

Received: 03 November 2023

Accepted: 24 June 2024

Accepted Manuscript online:
25 June 2024

Article published online:
25 September 2024

• References

• 1 Bolton-Maggs PHB, Watt A. Transfusion errors—can they be eliminated?. Br J Haematol 2020; 189 (01) 9-20
• 2 Bachowski G, Borge D, Brunker P, Eder A, Fialkow L, Fridey J. A Compendium of Transfusion Practice Guidelines. Washington, DC: American Red Cross; 2017
• 3 Yarahuan JWB, Billet A, Hron JD. A quality improvement initiative to decrease platelet ordering errors and a proposed model for evaluating clinical decision support effectiveness. Appl Clin Inform 2019; 10 (03) 505-512
• 4 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
• 5 Morse B, Anstett T, Mistry N. et al. User-centered design to reduce inappropriate blood transfusion orders. Appl Clin Inform 2023; 14 (01) 28-36
• 6 Schneider T, Jackups Jr R. Transfusion medicine informatics: a review of current practice and a glimpse into the future. Clin Lab Med 2021; 41 (04) 713-725
• 7 Sanderson B, Field JD, Kocaballi AB. et al. Multicenter, multidisciplinary user-centered design of a clinical decision-support and simulation system for massive transfusion. Transfusion 2023; 63 (05) 993-1004
• 8 Sanderson BJ, Field JD, Kocaballi AB. et al. Clinical decision support versus a paper-based protocol for massive transfusion: impact on decision outcomes in a simulation study. Transfusion 2023; 63 (12) 2225-2233
• 9 Erik H. FRAM: The Functional Resonance Analysis Method: Modelling Complex Socio-Technical Systems. CRC Press; 2017
• 10 Carayon P, Schoofs Hundt A, Karsh BT. et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 (suppl 1, suppl 1): i50-i58
• 11 Kandaswamy S, Jones J, Laporte D, Orenstein E. Combining lean methodology with functional resonance analysis method to understand blood product administration. Proc Hum Factors Ergon Soc Annu Meet 2022; 66: 766-768
• 12 Motulsky SL. Is member checking the gold standard of quality in qualitative research?. Qual Psychol 2021; 8: 389-406
• 13 Brooke J. SUS: a quick and dirty usability scale. Usability Eval Ind 1995; 189-194
• 14 Yen P-Y, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422
• 15 Arkley J, Ong LP, Gunaratnam N, Butt T, Clark SC. A distractions capture tool for cardiac surgery and lung transplantation: impact on outcomes. J Cardiothorac Surg 2023; 18 (01) 46
• 16 Liu D, Grundgeiger T, Sanderson PM, Jenkins SA, Leane TA. Interruptions and blood transfusion checks: lessons from the simulated operating room. Anesth Analg 2009; 108 (01) 219-222
• 17 Mann DM, Chokshi SK, Kushniruk A. Bridging the gap between academic research and pragmatic needs in usability: a hybrid approach to usability evaluation of health care information systems. JMIR Human Factors 2018; 5 (04) e10721