Subscribe to RSS
![](/products/assets/desktop/img/oa-logo.png)
DOI: 10.1055/s-0041-1740919
Agile, Easily Applicable, and Useful eHealth Usability Evaluations: Systematic Review and Expert-Validation
![](https://www.thieme-connect.de/media/10.1055-s-00035026/202201/lookinside/thumbnails/10-1055-s-0041-1740919_210136ra-1.jpg)
Abstract
Background Electronic health (eHealth) usability evaluations of rapidly developed eHealth systems are difficult to accomplish because traditional usability evaluation methods require substantial time in preparation and implementation. This illustrates the growing need for fast, flexible, and cost-effective methods to evaluate the usability of eHealth systems. To address this demand, the present study systematically identified and expert-validated rapidly deployable eHealth usability evaluation methods.
Objective Identification and prioritization of eHealth usability evaluation methods suitable for agile, easily applicable, and useful eHealth usability evaluations.
Methods The study design comprised a systematic iterative approach in which expert knowledge was contrasted with findings from literature. Forty-three eHealth usability evaluation methods were systematically identified and assessed regarding their ease of applicability and usefulness through semi-structured expert interviews with 10 European usability experts and systematic literature research. The most appropriate eHealth usability evaluation methods were selected stepwise based on the experts' judgements of their ease of applicability and usefulness.
Results Of these 43 eHealth usability evaluation methods identified as suitable for agile, easily applicable, and useful eHealth usability evaluations, 10 were recommended by the experts based on their usefulness for rapid eHealth usability evaluations. The three most frequently recommended eHealth usability evaluation methods were Remote User Testing, Expert Review, and Rapid Iterative Test and Evaluation Method. Eleven usability evaluation methods, such as Retrospective Testing, were not recommended for use in rapid eHealth usability evaluations.
Conclusion We conducted a systematic review and expert-validation to identify rapidly deployable eHealth usability evaluation methods. The comprehensive and evidence-based prioritization of eHealth usability evaluation methods supports faster usability evaluations, and so contributes to the ease-of-use of emerging eHealth systems.
Keywords
telemedicine and telehealth - clinical research informatics - clinical information systems - human–computer interaction - hospital information systemsProtection of Human and Animal Subjects
Ethical approval was not required for this study.
Publication History
Received: 24 May 2021
Accepted: 20 October 2021
Article published online:
09 March 2022
© 2022. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)
Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany
-
References
- 1 Lau F, Kuziemsky CE. eds. Handbook of eHealth Evaluation: An Evidence-Based Approach. Victoria, British Columbia Canada: University of Victoria; 2016
- 2 International Telecommunication Union (ITU). Standardization in e-health. Accessed August 9, 2021 at: https://www.itu.int/itunews/issue/2003/06/standardization.html
- 3 World Health Organization (WHO). eHealth. Accessed August 9, 2021 at: https://www.euro.who.int__data/assets/pdf_file/0010/261694/6.-eHealth,-Factsheet-for-European-Parliament.pdf
- 4 Bockhacker M, Syrek H, Elstermann von Elster M, Schmitt S, Roehl H. Evaluating usability of a touchless image viewer in the operating room. Appl Clin Inform 2020; 11 (01) 88-94
- 5 Richter Lagha R, Burningham Z, Sauer BC. et al. Usability testing a potentially inappropriate medication dashboard: a core component of the dashboard development process. Appl Clin Inform 2020; 11 (04) 528-534
- 6 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
- 7 Hron JD, Parsons CR, Williams LA, Harper MB, Bourgeois FC. Rapid implementation of an inpatient telehealth program during the COVID-19 pandemic. Appl Clin Inform 2020; 11 (03) 452-459
- 8 Rödle W, Wimmer S, Zahn J. et al. User-centered development of an online platform for drug dosing recommendations in pediatrics. Appl Clin Inform 2019; 10 (04) 570-579
- 9 Farzandipour M, Nabovati E, Heidarzadeh Arani M, Akbari H, Sharif R, Anvari S. Enhancing asthma patients' self-management through smartphone-based application: design, usability evaluation, and educational intervention. Appl Clin Inform 2019; 10 (05) 870-878
- 10 Klaassen B, van Beijnum BJ, Hermens HJ. Usability in telemedicine systems-a literature survey. Int J Med Inform 2016; 93: 57-69
- 11 Kushniruk AW, Borycki EM. Low-cost rapid usability engineering: designing and customizing usable healthcare information systems. Healthc Q 2006; 9 (04) 98-102
- 12 Ali H, Cole A, Panos G. Transforming patient hospital experience through smart technologies. In: Proceedings of the International Conference on Human-Computer Interaction; HCII '20. Copenhagen, Denmark: Springer International Publishing; 2020: 203-215
- 13 Coppersmith NA, Sarkar IN, Chen ES. Quality informatics: the convergence of healthcare data, analytics, and clinical excellence. Appl Clin Inform 2019; 10 (02) 272-277
- 14 Kushniruk A, Borycki E. Low-cost rapid usability testing: its application in both product development and system implementation. Stud Health Technol Inform 2017; 234: 195-200
- 15 Price M, Weber J, Bellwood P, Diemert S, Habibi R. Evaluation of eHealth system usability and safety. In: Handbook of EHealth Evaluation: An Evidece-Based Approach. Victoria, British Columbia Canada: University of Victoria; 2016: 337-350
- 16 Kushniruk AW, Borycki EM. Integrating low-cost rapid usability testing into agile system development of healthcare IT: a methodological perspective. Stud Health Technol Inform 2015; 210: 200-204
- 17 Marcilly R, Schiro J, Beuscart-Zéphir MC, Magrabi F. Building usability knowledge for health information technology: a usability-oriented analysis of incident reports. Appl Clin Inform 2019; 10 (03) 395-408
- 18 Broekhuis M, van Velsen L, Hermens H. Assessing usability of eHealth technology: a comparison of usability benchmarking instruments. Int J Med Inform 2019; 128: 24-31
- 19 International Organization for Standardization (ISO). Ergonomics of human-system interaction - Part 11: usability: definitions and concepts. Accessed July 26, 2021 at: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
- 20 Paelke V, Röcker C. User interfaces for cyber-physical systems: challenges and possible approaches. In: Proceedings of the 6th International Conference on Design, User Experience, and Usability; DUXU '17. Vancouver, BC, Canada: Springer International Publishing; 2017. ;10288: 75-85
- 21 Bastien JM. Usability testing: a review of some methodological and technical aspects of the method. Int J Med Inform 2010; 79 (04) e18-e23
- 22 Speicher M, Both A, Gaedke M. INUIT: the interface usability instrument. In: Proceedings of the 6th International Conference on Design, User Experience, and Usability; DUXU '17. Vancouver, BC, Canada: Springer International Publishing; 2017. ;10288: 256-268
- 23 Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019; 126: 95-104
- 24 Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78 (05) 340-353
- 25 Thompson KE, Rozanski EP, Haake AR. Here, there, anywhere: remote usability testing that works. In: Proceedings of the 5th Conference on Information Technology Education; CITC5 '04. Salt Lake City, UT, USA: ACM; 2004: 132-137
- 26 Wysocki T, Pierce J, Caldwell C. et al. A web-based coping intervention by and for parents of very young children with type 1 diabetes: user-centered design. JMIR Diabetes 2018; 3 (04) e16
- 27 González Sánchez JL, Padilla Zea N, Gutiérrez FL. From usability to playability: introduction to player-centred video game development process. In: Proceedings of the International Conference on Human Centered Design; HCD '09. San Diego, CA, USA: Springer; 2009. ;5619: 65-74
- 28 Dubey A, Singi K, Kaulgud V. Personas and redundancies in crowdsourced testing. In: Proceedings of the 12th International Conference on Global Software Engineering; ICGSE '17. Buenos Aires, Argentinia: IEEE; 2017: 76-80
- 29 Silva T, Silveira M, Maurer F. Usability evaluation practices within agile development. In: Proceedings of the 48th Hawaii International Conference on System Sciences. HICSS '15. Kauai, HI, USA: IEEE; 2015: 5133-5142
- 30 Pawson M, Greenberg S. Extremely rapid usability testing. J Usability Stud 2009; 4: 124-135
- 31 Kane D. Finding a place for discount usability engineering in agile development: throwing down the gauntlet. In: Proceedings of the Agile Development Conference; ADC '03. Salt Lake City, UT, USA: IEEE; 2003: 40-46
- 32 Grigoreanu V, Mohanna M. Informal Cognitive Walkthroughs (ICW): paring down and pairing up for an agile world. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '13. Paris, France: ACM; 2013: 3093-3096
- 33 Marques AB, Figueiredo R, Amorin W, Rabelo J, Barbosa SDJ, Conte T. Do usability and agility combine?: investigating the adoption of usability modeling in an agile software project in the industry. In: Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems; IHC '18. Belém, Brazil: ACM; 2018: 1-11
- 34 Cavichi de Freitas R, Rodrigues LA, Marques da Cunha A. AGILUS: a method for integrating usability evaluations on agile software development. In: Proceedings of the International Conference on Human-Computer Interaction; HCI '16. Cham, Switzerland: Springer International Publishing; 2016. ;9731: 545-552
- 35 Magües D, Castro J, Acuna S. HCI usability techniques in agile development. In: Proceedings of the International Conference on Automatica; ICA-ACCA '16. Curico, Chile: IEEE; 2016: 1-7
- 36 Butt S, Onn A, Butt M, Inam N, Butt S. Incorporation of usability evaluation methods in agile software model. In: Proceedings of the 17th IEEE International Multi Topic Conference; INMIC '14. Karachi, Pakistan: IEEE; 2014: 193-199
- 37 Federoff M, Villamor C, Miller L. et al. Extreme usability: adapting research approaches for agile development. In: Proceedings of the 26th Annual CHI Conference Extended Abstracts on Human Factors in Computing Systems; CHI EA '08. Florence, Italy: ACM; 2008: 2269-2272
- 38 Nielsen J. Discount usability: 20 years. Accessed September 13, 2019 at: https://www.nngroup.com/articles/discount-usability-20-years/
- 39 Schwartz L. Agile-user experience design: an agile and user-centered process?. In: Proceedings of the 8th International Conference on Software Engineering Advances; ICSEA '13. Venice, Italy: IARIA; 2013: 346-351
- 40 Isa W, Lokman A, Aris S. et al. Engineering rural informatics using agile user centered design. In: Proceedings of the 2nd International Conference on Information and Communication Technology; ICoICT '14. Bandung, Indonesia: IEEE; 2014: 367-372
- 41 Constantine LL, Lockwood LAD. Usage-centered software engineering: an agile approach to integrating users, user interfaces, and usability into software engineering practice. In: Proceedings of the 25th International Conference on Software Engineering; ICSE '03. Portland, OR, USA: IEEE; 2003: 746-747
- 42 Verhoeven F, van Gemert-Pijnen J. Discount user-centered e-health design: a quick-but-not-dirty method. In: Proceedings of the Symposium of the Austrian HCI and Usability Engineering Group; USAB '10. Klagenfurt, Austria: Springer; 2010. 638. 101-123
- 43 Russ AL, Baker DA, Fahner WJ. et al. A rapid usability evaluation (RUE) method for health information technology. AMIA Annu Symp Proc 2010; 2010: 702-706
- 44 European Union Agency for Network and Information Security (ENISA). Security and Resilience in eHealth: Security Challenges and Risks. Iraklio, Greece: ENISA; 2015: 1-48
- 45 Kuziemsky CE, Kushniruk A. Context mediated usability testing. Stud Health Technol Inform 2014; 205: 905-909
- 46 Øvad T, Larsen L. The prevalence of UX design in agile development processes in industry. In: Proceedings of the Agile Conference; AGILE '15. Washington, DC, USA: IEEE; 2015: 40-49
- 47 Duh HB-L, Tan GCB, Chen VH. Usability evaluation for mobile device: a comparison of laboratory and field tests. In: Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services; MobileHCI '06. Helsinki, Finland: ACM; 2006: 181-186
- 48 Whitenton K. Tools for unmoderated usability testing. Accessed April 28, 2021 at: https://www.nngroup.com/articles/unmoderated-user-testing-tools/
- 49 Nielsen J. Usability inspection methods. In: Proceedings of the Conference Companion on Human Factors in Computing Systems; CHI '94. Boston, MA, USA: ACM; 1994: 413-414
- 50 Johnson CM, Johnston D, Crowley PK. et al. EHR Usability Toolkit: A Background Report on Usability and Electronic Health Records. Accessed August 8, 2021 at: https://digital.ahrq.gov/sites/default/files/docs/citation/EHR_Usability_Toolkit_Background_Report.pdf
- 51 Davis R, Gardner J, Schnall R. A review of usability evaluation methods and their use for testing eHealth HIV interventions. Curr HIV/AIDS Rep 2020; 17 (03) 203-218
- 52 Wu DTY, Vennemeyer S, Brown K. et al. Usability testing of an interactive dashboard for surgical quality improvement in a large congenital heart center. Appl Clin Inform 2019; 10 (05) 859-869
- 53 Elbabour F, Alhadreti O, Mayhew P. Eye tracking in retrospective think-aloud usability testing: is there added value?. J Usability Stud 2017; 12 (03) 95-110
- 54 Elling S, Lentz L, de Jong M. Retrospective think-aloud method: using eye movements as an extra cue for participants' verbalizations. In: Proceedings of the 29th Annual CHI Conference on Human Factors in Computing Systems; SIGCHI '11. Vancouver, BC, Canada: ACM; 2011: 1161-1170
- 55 Khajouei R, Zahiri Esfahani M, Jahani Y. Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems. J Am Med Inform Assoc 2017; 24 (e1): e55-e60
- 56 Georgsson M, Staggers N, Årsand E, Kushniruk A. Employing a user-centered cognitive walkthrough to evaluate a mHealth diabetes self-management application: a case study and beginning method validation. J Biomed Inform 2019; 91: 1-15
- 57 Ghalibaf AK, Jangi M, Habibi MRM, Zangouei S, Khajouei R. Usability evaluation of obstetrics and gynecology information system using cognitive walkthrough method. Electron Physician 2018; 10 (04) 6682-6688
- 58 Rowley DE, Rhoades DG. The cognitive jogthrough: a fast-paced user interface evaluation procedure. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; CHI '92. Monterey, Calif, USA: ACM; 1992: 389-395
- 59 Andersen SB, Rasmussen CK, Frøkjær E. Bringing content understanding into usability testing in complex application domains - a case study in eHealth. In: Proceedings of the 6th International Conference on Design, User Experience, and Usability; DUXU '17. Vancouver, BC, Canada: Springer International Publishing; 2017. ;10288: 327-341
- 60 The User Experience Professionals' Association (UXPA). Usability body of knowledge. Accessed August 9, 2021 at: http://www.usabilitybok.org/methods
- 61 University of Minnesota Duluth (UMD). Usability evaluation toolbox. Accessed August 15, 2021 at: https://www.d.umn.edu/itss/training/online/usability/toolbox.html
- 62 Maguire M. Methods to support human-centred design. Int J Hum Comput Stud 2001; 55 (04) 587-634
- 63 Frøkjær E, Hornbæk K. Cooperative usability testing. In: Proceedings of CHI Extended Abstracts on Human Factors in Computing Systems; CHI EA '05. Portland, OR, USA: ACM; 2005: 1383-1386
- 64 Power C, Petrie H, Mitchell R. A framework for remote user evaluation of accessibility and usability of websites. In: Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction; UAHCI '09. San Diego, CA, USA: Springer; 2009. ;5614: 594-601
- 65 Hammontree M, Weiler P, Nayak N. Remote usability testing. Interact 1994; 1 (03) 21-25
- 66 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37 (01) 56-76
- 67 Medlock MC, Wixon D, McGee M, Welsh D. The rapid iterative test and evaluation method. In: Int Techn, Cost-Justifying Usability. 2nd ed.. Elsevier; 2005: 489-517
- 68 Wilson C. User Interface Inspection Methods: A User-Centered Design Method. Amsterdam, The Netherlands; Elsevier. 2013
- 69 Laemller R. Guerrilla Testing: Muting und rasch zu Resultaten. Accessed November 2, 2020 at: https://www.testingtime.com/blog/guerrilla-testing/
- 70 Firmenich S, Garrido A, Grigera J, Rivero JM, Rossi G. Usability improvement through A/B testing and refactoring. Softw Qual J 2018; 27 (01) 203-240
- 71 Guaiani F, Muccini H. Crowd and laboratory testing, can they co-exist? An exploratory study. In: Proceedings of the 2nd International Workshop on CrowdSourcing in Software Engineering; CSI-SE '15. Florence, Italy: IEEE; 2015: 32-37
- 72 Zogaj S, Bretschneider U, Leimeister JM. Managing crowdsourced software testing: a case study based insight on the challenges of a crowdsourcing intermediary. J Bus Econ 2014; 84 (03) 375-405
- 73 Downey LL. Group usability testing: evolution in usability techniques. J Usability Stud 2007; 2 (03) 133-144
- 74 Sears A. Heuristic walkthroughs: finding the problems without the noise. Int J Hum Comput Interact 1997; 9 (03) 213-234
- 75 Nielsen L, Madsen S. The usability expert's fear of agility: an empirical study of global trends and emerging practices. In: Proceedings of the 7th Nordic Conference on Human-Computer Interaction; NordiCHI '12. Copenhagen, Denmark: ACM; 2012: 261-264
- 76 Höysniemi J, Hämäläinen P, Turkki L. Using peer tutoring in evaluating the usability of a physically interactive computer game with children. Interact Comput 2003; 15 (02) 203-255
- 77 Khalayli N, Nyhus S, Hamnes K, Terum T. Persona based rapid usability kick-off. In: Proceedings of CHI Extended Abstracts on Human Factors in Computing Systems; CHI EA '07. San Jose, CA, USA: ACM; 2007: 1771-1776
- 78 Usability in Germany eV (UIG). Fokusgruppe. Accessed November 9, 2020 at: https://www.usability-in-germany.de/definition/fokusgruppe
- 79 Rosenbaum S, Cockton G, Coyne K, Muller M, Rauch T. Focus groups in HCI. In: Proceedings of CHI Extended Abstracts on Human Factors in Computing Systems; CHI EA '02. Minneapolis, MN, USA: ACM; 2002: 702-703
- 80 Usability in Germany eV (UIG). Fragebogen. Accessed November 9, 2020 at: https://www.usability-in-germany.de/definition/fragebogen
- 81 Krause R. Storyboards help visualize UX ideas. Nielsen Norman Group. Accessed November 9, 2020 at: https://www.nngroup.com/articles/storyboards-visualize-ideas/
- 82 Piyush J, Sanjay KD, Ajay R. Software usability evaluation method. Int J Adv Res Comput Eng Technol 2012; 1 (02) 28-33
- 83 Holzinger A, Slany WXP. + UE → XU Praktische Erfahrungen mit eXtreme Usability. Informatik-Spektrum 2006; 29 (02) 91-97
- 84 Borycki E, Senathirajah Y, Kushniruk AW. The future of mobile usability, workflow and safety testing. Stud Health Technol Inform 2017; 245: 15-19
- 85 Holzinger A. Usability engineering methods for software developers. Commun ACM 2005; 48 (01) 71-74