User Interface Problems of a Nationwide Inpatient Information System: A Heuristic Evaluation

Summary

Introduction

While studies have shown that usability evaluation could uncover many design problems of health information systems, the usability of health information systems in developing countries using their native language is poorly studied. The objective of this study was to evaluate the usability of a nationwide inpatient information system used in many academic hospitals in Iran.

Material and Methods

Three trained usability evaluators independently evaluated the system using Nielsen’s 10 usability heuristics. The evaluators combined identified problems in a single list and independently rated the severity of the problems. We statistically compared the number and severity of problems identified by HIS experienced and non-experienced evaluators.

Results

A total of 158 usability problems were identified. After removing duplications 99 unique problems were left. The highest mismatch with usability principles was related to “Consistency and standards” heuristic (25%) and the lowest related to “Flexibility and efficiency of use” (4%). The average severity of problems ranged from 2.4 (Major problem) to 3.3 (Catastrophe problem). The experienced evaluator with HIS identified significantly more problems and gave higher severities to problems (p<0.02).

Discussion

Heuristic Evaluation identified a high number of usability problems in a widely used inpatient information system in many academic hospitals. These problems, if remain unsolved, may waste users’ and patients’ time, increase errors and finally threaten patient’s safety. Many of them can be fixed with simple redesign solutions such as using clear labels and better layouts. This study suggests conducting further studies to confirm the findings concerning effect of evaluator experience on the results of Heuristic Evaluation.

Keywords

Hospital information systems - health information systems - user-computer interface - inpatients - usability - Heuristic Evaluation

 

Conflicts of Interest

The authors declare that there is no conflicts of interest in the research.

• References

• 1 Menachemi N, Collum TH. Benefits and drawbacks of electronic health record systems. Risk management and healthcare policy 2011; 04: 47-55.
• 2 Gulcin SCYucela, Bo Hoegec, Ahmet F. Ozok A fuzzy risk assessment model for hospital information sys- tem implementation. Expert Systems with Applications 2012; 39 (01) 1211-1218.
• 3 Ahmadian L, Salehi SNejad, Khajouei R. Evaluation methods used on health information systems (HISs) in Iran and the effects of HISs on Iranian healthcare: A systematic review. International journal of medical informatics 2015; 84 (06) 444-453.
• 4 Khajouei R, Jaspers MW. The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders: a systematic review. Methods of information in medicine 2010; 49 (01) 3-19.
• 5 HR. EHR History in Iran: Ministry of Health and Medical Education. 2009
• 6 JN. Usability Engineering. USA, San Diego: Academic Press; 1993
• 7 Khajouei R, Peek N, Wierenga PC, Kersten MJ, Jaspers MW. Effect of predefined order sets and usability problems on efficiency of computerized medication ordering. International journal of medical informatics 2010; 79 (10) 690-698.
• 8 Lathan CE, Sebrechts MM, Newman DJ, Doarn CR. Heuristic Evaluation of a web-based interface for internet telemedicine. Telemedicine journal: the official journal of the American Telemedicine Association 1999; 05 (02) 177-185.
• 9 Hertzum M. Images of Usability. International Journal of Human-Computer Interaction 2010; 26 (06) 567-600.
• 10 Choi J, Bakken S. Web-based education for low-literate parents in Neonatal Intensive Care Unit: development of a website and Heuristic Evaluation and usability testing. International journal of medical informatics 2010; 79 (08) 565-575.
• 11 Turner-Bowker DM, Saris-Baglama RN, Smith KJ, DeRosa MA, Paulsen CA, Hogue SJ. Heuristic Evaluation and usability testing of a computerized patient-reported outcomes survey for headache sufferers. Telemedicine journal and e-health : the official journal of the American Telemedicine Association 2011; 17 (01) 40-45.
• 12 Nielsen JMR. Usability Inspection Methods. New York: John Wiley & Sons; 1994
• 13 Molich NJIah-cdR. Improving a human-computer dialogue: Commun ACM. 1990; 338-348.
• 14 Tang Z, Johnson TR, Tindall RD, Zhang J. Applying Heuristic Evaluation to improve the usability of a telemedicine system. Telemedicine journal and e-health : the official journal of the American Telemedicine Association 2006; 12 (01) 24-34.
• 15 Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. Journal of biomedical informatics 2003; 36 (1–2): 23-30.
• 16 Thyvalikakath TP, Monaco V, Thambuganipalle H, Schleyer T. Comparative study of Heuristic Evaluation and usability testing methods. Studies in health technology and informatics 2009; 143: 322-327.
• 17 Thyvalikakath TP, Schleyer TK, Monaco V. Heuristic Evaluationofclinicalfunctionsinfourpractice- managementsystems: a pilot study. J Am Dent Assoc 2007; 138 (02) 208-212.
• 18 Joshi A, Arora M, Dai L, Price K, Vizer L, Sears A. Usability of a patient education and motivation tool using Heuristic Evaluation. Journal of medical Internet research 2009; 11 (04) e47.
• 19 Kimiafar KMG, Sadoughi F, Hosseini F. A study on the user’s views on the quality of teaching hospitals information system of Mashhad University of Medical Sciences-2006. Journal of Health Administration 2007; 10 (29) 31-36.
• 20 Nabovati E, Vakili-Arki H, Eslami S, Khajouei R. Usability evaluation of Laboratory and Radiology Information Systems integrated into a hospital information system. Journal of medical systems 2014; 38 (04) 35.
• 21 Khajouei RAA, Atashi A. Usability Evaluation of an Emergency Information System: A Heuristic Evaluation. Journal of Health Administration 2013; 16 (25) 61-72.
• 22 Chan AJ, Islam MK, Rosewall T, Jaffray DA, Easty AC, Cafazzo JA. Applying usability heuristics to radio-therapy systems. Radiotherapy and oncology: journal of the European Society for Therapeutic Radiology and Oncology 2012; 102 (01) 142-147.
• 23 Khajouei RAA, Atashi A. Usability Evaluation of an Emergency Information System: A Heuristic Evaluation. Journal of Health Administration 2013; 16 (25) 61-72.
• 24 Edwards PJMKP, Jacko JA, Sainfort F. Evaluating usability of a commercial electronic healthrecord: Acase study. Int J Hum Comput Stud 2008; 66 (10) 718-728.
• 25 Nielsen J. How to Conduct a Heuristic Evaluation1995 22 September 2015. Available from: http://www.ngroup.com/articles/how-to-conduct-a-heuristic-evaluation/
• 26 Ahmadian LCR, Van Klei WA, DE Keizer NF. Data collection variation in preoperative assessment: a literature review. Comput Inform Nurs 2011; 29 (11) 662-670.
• 27 Ahmadian LCR, Van Klei WA, DE Keizer NF. Diversity in preoperative-assessment data collection, a literature review. Stud Health Technol Inform 2008; 136: 127-132.
• 28 Hertzum MJN. The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods. International Journal of Human-Computer Interaction. 2003 15. (1)
• 29 Penha M, Correia WFM, Campos FFdC, Barros MdLN. Heuristic Evaluation of Usability – a Case study with the Learning Management Systems (LMS) of IFPE. International Journal of Humanities and Social Science 2014; 04: 295-303.
• 30 Khajouei R, de Jongh D, Jaspers MW. Usability evaluation of a computerized physician order entry for medication ordering. Studies in health technology and informatics 2009; 150: 532-536.
• 31 Cockton G, Woolrych A. Understanding inspection methods: lessons from an assessment of Heuristic Evaluation. In People and Computers XV – Interaction without Frontiers Springer London 2001; 171-191.
• 32 Cockton G, Woolrych A. Sale must end: should discount methods be cleared off HCI’s shelves?. Interactions 2002; 09 (05) 13-18.
• 33 Law E, Hvannberg E. Analysis of strategies for improving and estimating the effectiveness of Heuristic Evaluation. The third Nordic conference on Human-computer interaction. 2004. ACM.
• 34 Morten RMHertzum, Niels JacobsenEbbe. What you get is what you see: revisiting the evaluator effect in usability tests. Behaviour & Information Technology 2014; 33 (02) 144-163.
• 35 Khajouei RAL, Jaspers MW. Methodological concerns in usability evaluation of software prototypes. J Biomed Inform 2011; 44 (04) 700-701.
• 36 Hornbæk K. Dogmas in the assessment of usability evaluation methods. Behaviour & Information Technology 2010; 29 (01) 97-111.