Formative Usability Testing Reduces Severe Blood Product Ordering Errors

Abstract

Background Medical errors in blood product orders and administration are common, especially for pediatric patients. A failure modes and effects analysis in our health care system indicated high risk from the electronic blood ordering process.

Objectives There are two objectives of this study as follows:

(1) To describe differences in the design of the original blood product orders and order sets in the system (original design), new orders and order sets designed by expert committee (DEC), and a third-version developed through user-centered design (UCD).

(2) To compare the number and type of ordering errors, task completion rates, time on task, and user preferences between the original design and that developed via UCD.

Methods A multidisciplinary expert committee proposed adjustments to existing blood product order sets resulting in the DEC order set. When that order set was tested with front-line users, persistent failure modes were detected, so orders and order sets were redesigned again via formative usability testing. Front-line users in their native clinical workspaces were observed ordering blood in realistic simulated scenarios using a think-aloud protocol. Iterative adjustments were made between participants. In summative testing, participants were randomized to use the original design or UCD for five simulated scenarios. We evaluated differences in ordering errors, time on task, and users' design preference with two-sample t-tests.

Results Formative usability testing with 27 providers from seven specialties led to 18 changes made to the DEC to produce the UCD. In summative testing, error-free task completion for the original design was 36%, which increased to 66% in UCD (30%, 95% confidence interval [CI]: 3.9–57%; p = 0.03). Time on task did not vary significantly.

Conclusion UCD led to substantially different blood product orders and order sets than DEC. Users made fewer errors when ordering blood products for pediatric patients in simulated scenarios when using the UCD orders and order sets compared with the original design.

Keywords

electronic health records - blood transfusion - clinical decision support - usability - human–computer interaction

Protection of Human and Animal Subjects

This work was felt to be primarily focused on quality improvement and therefore deemed nonhuman subjects research by the Institutional Review Board of Children's Healthcare of Atlanta.

• References

• 1 Rowley M. Errors related to information technology (IT). In: Bolton-Maggs P. , ed. Annual SHOT Report: 2017. Manchester, England: Serious Hazards of Transfusion (SHOT). 2018: 104-105
• 2 New HV, Berryman J, Bolton-Maggs PH. , et al; British Committee for Standards in Haematology. Guidelines on transfusion for fetuses, neonates and older children. Br J Haematol 2016; 175 (05) 784-828
• 3 Goel R, Cushing MM, Tobian AAR. Pediatric patient blood management programs: not just transfusing little adults. Transfus Med Rev 2016; 30 (04) 235-241
• 4 Zantek ND, Parker RI, van de Watering LM. , et al; Pediatric Critical Care Transfusion and Anemia Expertise Initiative (TAXI); Pediatric Critical Care Blood Research Network (BloodNet), and the Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network. Recommendations on selection and processing of RBC components for pediatric patients from the pediatric critical care transfusion and anemia expertise initiative. Pediatr Crit Care Med 2018; 19 (9S, Suppl 1): S163-S169
• 5 Jones H, Reeve K. Transfusion guidelines in children: II. Anaesth Intensive Care Med 2017; 18: 546-550
• 6 Rothschild JM, McGurk S, Honour M. , et al. Assessment of education and computerized decision support interventions for improving transfusion practice. Transfusion 2007; 47 (02) 228-239
• 7 McGreevey III JD. Order sets in electronic health records: principles of good practice. Chest 2013; 143 (01) 228-235
• 8 Yarahuan JW, Billet A, Hron JD. A quality improvement initiative to decrease platelet ordering errors and a proposed model for evaluating clinical decision support effectiveness. Appl Clin Inform 2019; 10 (03) 505-512
• 9 Institute for Healthcare Improvement. Quality improvement essentials toolkit. Available at: http://www.ihi.org/resources/Pages/Tools/Quality-Improvement-Essentials-Toolkit.aspx . Accessed December 4, 2019
• 10 Ahmadian L, Khajouei R. Impact of computerized order sets on practitioner performance. Stud Health Technol Inform 2012; 180: 1129-1131
• 11 Chan AJ, Chan J, Cafazzo JA. , et al. Order sets in health care: a systematic review of their effects. Int J Technol Assess Health Care 2012; 28 (03) 235-240
• 12 Gartner D, Zhang Y, Padman R, Gartner gartnerd D. Cognitive workload reduction in hospital information systems decision support for order set optimization. Health Care Manage Sci 2018; 21 (02) 224-243
• 13 Kara A, Isaacs AN, Nisly SA. Prescriptions for bedtime sedatives after the introduction of a general admission order set at an academic health center: The potential and pitfalls of order sets. J Patient Saf 2017; 13 (04) 232-236
• 14 Leu MG, Morelli SA, Chung O-Y, Radford S. Systematic update of computerized physician order entry order sets to improve quality of care: a case study. Pediatrics 2013; 131 (Suppl. 01) S60-S67
• 15 Osheroff JA, Teich J, Levick D. , et al. Improving Outcomes with Clinical Decision Support: An Implementer's Guide. 2nd ed. Chicago, IL: HIMSS Publishing; 2012
• 16 Greenes RA, Bates DW, Kawamoto K, Middleton B, Osheroff J, Shahar Y. Clinical decision support models and frameworks: Seeking to address research issues underlying implementation successes and failures. J Biomed Inform 2018; 78: 134-143
• 17 Li RC, Wang JK, Sharp C, Chen JH. When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record. BMJ Qual Saf 2019; 28 (12) 987-996
• 18 Melton BL, Zillich AJ, Russell SA. , et al. Reducing prescribing errors through creatinine clearance alert redesign. Am J Med 2015; 128 (10) 1117-1125
• 19 Schumacher RM, Lowry SZ, Locke G, Gallagher PD. NIST guide to the processes approach for improving the usability of electronic health records. Available at: https://www.nist.gov/system/files/documents/itl/hit/Guide_Final_Publication_Version.pdf . Accessed December 4, 2019
• 20 Ash JS, Smith III AC, Stavri PZ. Performing subjectivist studies in the qualitative traditions responsive to users. In: Friedman CP, Wyatt JC. , eds. Evaluation Methods in Medical Informatics. New York, NY: Springer-Verlag; 2006: 267-300
• 21 Russ AL, Saleem JJ. Ten factors to consider when developing usability scenarios and tasks for health information technology. J Biomed Inform 2018; 78: 123-133
• 22 Venkatesh V, Davis FD. A theoretical extension of the technology acceptance model: four longitudinal field studies. Manage Sci 2000; 46: 186-204
• 23 R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Available at: https://www.R-project.org/
• 24 Champely S, Ekstrom C, Dalgaard P. , et al. Package ‘pwr’: Basic Functions for Power Analysis. Available at: https://cran.r-project.org/web/packages/pwr/pwr.pdf . Accessed December 4, 2019
• 25 Horsky J, Ramelson HZ. Cognitive errors in reconciling complex medication lists. AMIA Annu Symp Proc 2017; 2016: 638-646
• 26 Reason J. Human error: models and management. BMJ 2000; 320 (7237): 768-770
• 27 Yu CH, Stephenson AL, Gupta S. The effect of patient care order sets on medical resident education: a prospective before-after study. BMC Med Educ 2013; 13: 146
• 28 Lewis Jr JB, Ryder K. Medical education and decision-support systems. Virtual Mentor 2011; 13 (03) 156-160
• 29 Miller CA. Lessons from another industry: aviation, usability, and medical device design. Biomed Instrum Technol 2013; 47 (Suppl): 40-44
• 30 Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Comput Biol Med 2002; 32 (03) 141-149