A Pilot Study on Usability Analysis of Emergency Department Information System by Nurses

Summary

Objectives: Employing new health information technologies while concurrently providing quality patient care and reducing risk is a major challenge in all health care sectors. In this study, we investigated the usability gaps in the Emergency Department Information System (EDIS) as ten nurses differentiated by two experience levels, namely six expert nurses and four novice nurses, completed two lists of nine scenario-based tasks.

Methods: Standard usability tests using video analysis, including four sets of performance measures, a task completion survey, the system usability scale (SUS), and sub-task analysis were conducted in order to analyze usability gaps between the two nurse groups.

Results: A varying degree of usability gaps were observed between the expert and novice nurse groups, as novice nurses completed the tasks both less efficiently, and expressed less satisfaction with the EDIS. The most interesting finding in this study was the result of ‘percent task success rate,’ the clearest performance measure, with no substantial difference observed between the two nurse groups. Geometric mean values between expert and novice nurse groups for this measure were 60% vs. 62% in scenario 1 and 66% vs. 55% in scenario 2 respectively, while there were some marginal to substantial gaps observed in other performance measures. In addition to performance measures and the SUS, sub-task analysis highlighted navigation pattern differences between users, regardless of experience level.

Conclusion: This study will serve as a baseline study for a future comparative usability evaluation of EDIS in other institutions with similar clinical settings.

Keywords

Electronic health records - emergency department information system - EDIS - usability engineering

 

Conflict of Interest

The authors of this paper do not have any financial and personal relationships with other people or organizations that could inappropriately influence this work.

• References

• 1 Gans D, Kralewski J, Hammons T, Dowd B. Medical groups’ adoption of electronic health records and information systems. Health Aff (Millwood) 2005; 24 (Suppl. 05) 1323-1333.
• 2 Bertman J, Skolnik N. Poor usability keeps EHR adoption rates low. Family Practice News 2010; May 1.
• 3 Technology NIoSa. Common industry specification for usability requirements NISTIR 7432. Gaithersburg, MD2007. Report No.: IUSR:CISU-R v0.90.
• 4 Kim MS, Mohrer D, Trusko B, Landrigan P, Elkin P. World Trade Center Medical Monitoring and Treatment Program: A clinical workflow analysis. AMIA Clinical Research Informatics Summit 2010; San Francisco, CA2010.
• 5 Beuscart-Zephir MC, Elkin P, Pelayo S, Beuscart R. The human factors engineering approach to biomedical informatics projects: state of the art, results, benefits and challenges. Yearb Med Inform 2007: 109-127.
• 6 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293 (10) 1197-1203.
• 7 Armijo D, McDonnell C, Werner K. Electronic health record usability: Interface design considerations. AHRQ Publication No. 09(10)-0091–2-EF. Rockville, MD: Agency for Healthcare Research and Quality 2009.
• 8 Armijo D, McDonnell C, Werner K. Electronic healthe record usability-evaluation and use case framework. AHRQ Publication No. 09(10)-0091–1-EF. Rockville, MD: Agency for Healthcare Research and Quality 2009.
• 9 Belden JL, Grayson R, Barnes J. Defining and testing EMR usability: Principles and proposed methods of EMR usability evaluation and rating. HIMSS EHR Usability Task Force 2009.
• 10 Kane LR. Electronic medical record survey results: Medscape exclusive readers’ choice. Medscape Business of Medicine 2009.
• 11 Edsall RL, Adler KG. The 2009 EHR user satisfaction survey responses from 2,012 family physicians. Family Practice Management. 2009 16(6).
• 12 McDonnell C, Werner K, Wendel L. Electronic health record usability vendor practices and perspectives. Rockville, MD: Agency for Healthcare Research and Quality; 2010
• 13 Smelcer J, Miller-Jacobs H, Kantrovich L. Usability of electronic medical records. Journal of Usability Studies 2009; 4 (02) 70-84.
• 14 Current State of Emergency Department Information Systems. InfoHealth Management Corp 2007.
• 15 Pallin D, Lahman M, Baumlin K. Information technology in emergency medicine residency-affiliated emergency departments. Acad Emerg Med 2003; 10 (08) 848-852.
• 16 Taylor TB. Information management in the emergency department. Emerg Med Clin North Am 2004; 22 (01) 241-257.
• 17 Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; May 144 (Suppl. 10) 742-752.
• 18 Kaushal R, Barker KN, Bates DW. How can information technology improve patient safety and reduce medication errors in children’s health care?. Arch Pediatr Adolesc Med 2001; 155 (09) 1002-1007.
• 19 Baumlin KM, Shapiro JS, Weiner C, Gottlieb B, Chawla N, Richardson LD. Clinical information system and process redesign improves emergency department efficiency. Jt Comm J Qual Patient Saf 2010; 36 (04) 179-185.
• 20 Shapiro JS, Baumlin KM, Chawla N, Genes N, Godbold J, Ye F, Richardson LD. Emergency department information system implementation and process redesign result in rapid and sustained financial enhancement at a large academic center. Acad Emerg Med 2010; 17 (05) 527-535.
• 21 Virzi RA. Refining the test phase of usability evaluation: how many subjects is enough?. Hum Factors 1992; 34 (04) 457-468.
• 22 Tullis T, Albert B. Issues-based metrics. Measuring the user experience. San Francisco: Morgan Kaufmann; 2008; 99-121.
• 23 Barnum C. The magic number 5: Is it enough for web testing?. Information Design Journal 2003; 11: 160-170.
• 24 Lewis JR. Evaluation of procedures for adjusting problm-discovery rates estimated from small samples. The International Journal of Human-Computer Interaction 1994; 13 (04) 445-479.
• 25 Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. Proceedings of the INTERACT ‘93 and CHI ‘93 conference on Human factors in computing systems; Amsterdam, The Netherlands. 169166: ACM; 1993: 206-213.
• 26 Lewis JR. Sample sizes for usability studies: additional considerations. Human Factors 1994; 36 (02) 368-378.
• 27 Sauro J, Dumas JS. Comparison of three one-question, post-task usability questionnaires. CHI. 2009 ; Boston, MA2009.
• 28 Brooke J. SUS –a quick and dirty usability scale. Usability Evaluation in Industry. London ; Bristol, Pa.: Taylor & Francis; 1996
• 29 Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction 2008; 24 (06) 574-594.
• 30 Lewis JR, Sauro J. The factor structure of the system usability scale. Proceedings of the 1st International Conference on Human Centered Design: Held as Part of HCI International 2009; San Diego, CA. 1601645: Springer-Verlag; 2009: 94-103.
• 31 Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies 2009; 4 (03) 114-123.
• 32 Lutes KD, Chang K, Baggili IM. editors. Diabetic e-Management System (DEMS). Information Technology: New Generations, 2006 ITNG 2006 Third International Conference on; 2006 10–12 April 2006.
• 33 Kastner M, Lottridge D, Marquez C, Newton D, Straus SE. Usability evaluation of a clinical decision support tool for osteoporosis disease management. Implementation science. : IS2010; 5: 96.
• 34 Suominen O, Hyvonen E, Viljanen K, Hukka E. Health Finland –a national semantic publishing network and portal for health information. Web Semant 2009; 7 (04) 287-297.
• 35 CCHIT 2011 Usability Testing Guide for Ambulatory EHR’s. Certification Commission for Health Information Technolog 2009.
• 36 Schumacher RM, Lowry SZ. NIST guide to the processes approach for improving the usability of electronic health records. National Institute of Standards and Technology,. U. S. Department of Commerce 2010 Contract No.: NISTIR 7741.
• 37 Cordes RE. The effects of running fewer subjects on time-on-task measures. International Journal of Human-Computer Interaction 1993; 5 (Suppl. 04) 393-403.
• 38 Eisenhart C, Deming L, Martin CS. On the aritmetic mean and median in small samples from the normal and certain non-normal populations. Annals of Mathematical Statistics. 1948: 599.
• 39 Anschuetz L. How does corporate culture affect the reporting of usability results? UPA 2004 Idea Markets [serial on the Internet]. 2004
• 40 Nielsen J. First rule of usability? Don‘t listen to users. 2001
• 41 Sauro J. Do users fail a task and still rate it as easy?. 2009; Available from: http://www.measuringusability./com/failed-sat.php.
• 42 Kuperman GJ, Teich JM, Gandhi TK, Bates DW. Patient safety and computerized medication ordering at Brigham and Women’s Hospital. Joint Commission Journal on Quality and Patient Safety 2001; 27 (10) 509-521.
• 43 Buller-Close K, Schriger DL, Baraff LJ. Heterogeneous effect of an emergency department expert charting system. Ann Emerg Med 2003; 41 (05) 644-652.
• 44 Staggers N, Jennings BM, Lasome CEM. A usability assessment of AHLTA in ambulatory clinics at a military medical center. Military Medicine 2010; 175 (07) 518-524.
• 45 Li Q, Douglasa S, Hundt AS, Carayona P. editors. A heuristic usability evaluation of a computerized provider order entry (CPOE) technology. IEA2006 Congress; 2006
• 46 Edwards PJ, Moloney KP, Jacko JA, Sainfort F. Evaluating usability of a commercial electronic health record: A case study. International Journal of Human-Computer Studies 2008; 66 (10) 718-728.
• 47 Nielsen J, Pernice K. Eyetracking web usability. Berkeley, CA.: New Riders; 2010