The Development of Heuristics for Evaluation of Dashboard Visualizations

Abstract

Background Heuristic evaluation is used in human–computer interaction studies to assess the usability of information systems. Nielsen's widely used heuristics, first developed in 1990, are appropriate for general usability but do not specifically address usability in systems that produce information visualizations.

Objective This article develops a heuristic evaluation checklist that can be used to evaluate systems that produce information visualizations. Principles from Nielsen's heuristics were combined with heuristic principles developed by prior researchers specifically to evaluate information visualization.

Methods We used nominal group technique to determine an appropriate final set. The combined existing usability principles and associated factors were distributed via email to a group of 12 informatics experts from a range of health care disciplines. Respondents were asked to rate each factor on its importance as an evaluation heuristic for visualization systems on a scale from 1 (definitely don't include) to 10 (definitely include). The distribution of scores for each item were calculated. A median score of ≥8 represented consensus for inclusion in the final checklist.

Results Ten of 12 experts responded with rankings and written comments. The final checklist consists of 10 usability principles (7 general and 3 specific to information visualization) substantiated by 49 usability factors. Three nursing informatics experts then used the checklist to evaluate a vital sign dashboard developed for home care nurses, using a task list designed to explore the full functionality of the dashboard. The experts used the checklist without difficulty, and indicated that it covered all major usability problems encountered during task completion.

Conclusion The growing capacity to generate and electronically process health data suggests that data visualization will be increasingly important. A checklist of usability heuristics for evaluating information visualization systems can contribute to assuring high quality in electronic data systems developed for health care.

Keywords

requirements analysis and design - clinical decision support - home health - dashboard - methodologies

Protection of Human and Animal Subjects

The study protocol was reviewed and approved by the Institutional Review Boards at Columbia University and the Visiting Nurse Service of New York.

• References

• 1 Andrews K. Evaluation comes in many guises. In: Proceedings of the 2008 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization (BELIV, 2008); Florence, Italy. New York: ACM; 2008
• 2 Hermawati S, Lawson G. Establishing usability heuristics for heuristics evaluation in a specific domain: is there a consensus?. Appl Ergon 2016; 56: 34-51
• 3 10 usability heuristics for interface design. Available at: https://www.nngroup.com/articles/ten-usability-heuristics/ . Accessed June 10, 2018
• 4 Nielsen J, Molich R. Heuristic evaluation of user interfaces. In: CHI '90 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Seattle, Washington. New York: ACM; 1990 :249–256
• 5 Pierotti D. Heuristic Evaluation - A System Checklist. Available at: ftp://ftp.cs.uregina.ca/pub/class/305/lab2/example-he.html . Accessed June 10, 2018
• 6 Tarrell A, Forsell C, Fruhling AL, Grinstein G, Borgo R, Scholtz J. Toward Visualization-Specific Heuristic Evaluation. In: BELIV '14 Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization Paris, France. New York: ACM; 2014 :110–117
• 7 Väätäjä H, Varsaluoma J, Heimonen T. , et al. Information visualization heuristics in practical expert evaluation. In: BELIV '16 Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization Baltimore, MD. New York: ACM; 2016 :36–43
• 8 Gonzalez-Holland E, Whitmer D, Moralez L, Mouloua M. Examination of the use of Nielsen's 10 usability heuristics & outlooks for the future. Proc Hum Factors Ergon Soc Annu Meet 2017; 61 (01) 1472-1475
• 9 Forsell C, Johansson J. An heuristic set for evaluation in information visualization. In: AVI '10 Proceedings of the International Conference on Advanced Visual Interfaces Roma, Italy. New York: ACM; 2010 :199–206
• 10 Zuk T, Schlesier L, Neumann P, Hancock MS, Carpendale S. Heuristics for information visualization evaluation. In: BELIV '06 Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization Venice, Italy. New York: ACM; 2006 :1–6
• 11 Freitas C, Luzzardi P, Cava R, Winckler M, Pimenta MS, Nedel L. Evaluating usability of information visualization techniques. In: Proceedings of the 5th Symposium on Human Factors in Computer Systems, IHC 2002; Brazil. Brazil: Brazilian Computer Society; 2002 :40–51
• 12 Delbecq AL, Van de Ven AH, Gustafson DH. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Glenview, IL: Scott Foresman; 1975
• 13 Delbecq AL, Van de Ven AH. A group process model for problem identification and program planning. J Appl Behav Sci 1971; 7 (04) 466-492
• 14 Amar R, Stasko J. A knowledge task-based framework for design and evaluation of information visualizations. In: Proceedings of IEEE Symposium on Information Visualization, 2004 INFOVIS 2004; Austin, TX, USA. Washington, DC: IEEE Computer Society; 2004
• 15 Scapin DL, Bastien JMC. Ergonomic criteria for evaluating the ergonomic quality of interactive system. Behav Inf Technol 1997; 16 (4–5): 220-231
• 16 Shneiderman B. The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings of the IEEE Symposium on Visual Languages; Boulder, CO, USA. Washington, DC: IEEE Computer Society Press; 1996 :336–343
• 17 Nielsen J. Enhancing the explanatory power of usability heuristics. In: CHI '94 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: April 24–28, 1994; Boston, MA. New York, NY: ACM; 1994 :152–158
• 18 Zuk T, Carpendale S. Theoretical analysis of uncertainty visualizations. Proc. SPIE 6060, Visualization and Data Analysis 2006, 606007. Doi: 10.1117/12.643631
• 19 Stone JV. Principles of Neural Information Theory: A Tutorial Introduction. Sheffield, UK: Sebtel Press; 2016
• 20 Dowding DW, Russell D, Onorato N, Merrill JA. Technology solutions to support care continuity in home care: a focus group study. J Healthc Qual 2017
• 21 Docherty SL, Vorderstrasse A, Brandon D, Johnson C. Visualization of multidimensional data in nursing science. West J Nurs Res 2016; 39 (01) 112-126
• 22 Schumacher R, Lowry S. NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records. Gaithersburg, MD: National Institute of Standards and Technology; 2010
• 23 Rind A, Wang T, Aigner W. , et al. Interactive information visualization to explore and query electronic health records. Foundations Trends Human Comp Interact 2013; 5 (03) 207-298
• 24 Wang TD, Wongsuphasawat K, Plaisant C, Shneiderman B. Extracting insights from electronic health records: case studies, a visual analytics process model, and design recommendations. J Med Syst 2011; 35 (05) 1135-1152
• 25 West VL, Borland D, Hammond WE. Innovative information visualization of electronic health record data: a systematic review. J Am Med Inform Assoc 2015; 22 (02) 330-339
• 26 Shneiderman B, Plaisant C, Hesse B. Improving healthcare with interactive visualization. Computer 2013; 46 (05) 58-66
• 27 National Library of Medicine (U.S.). Board of Regents: a platform for biomedical discovery and data-powered health: National Library of Medicine strategic plan 2017–202. In: NIH Publication. Bethesda, MD: National Institutes of Health, National Library of Medicine; 2017