Download PDF
Appl Clin Inform 2019; 10(05): 859-869
DOI: 10.1055/s-0039-1698466
Special Topic: Visual Analytics in Healthcare
Georg Thieme Verlag KG Stuttgart · New York

Usability Testing of an Interactive Dashboard for Surgical Quality Improvement in a Large Congenital Heart Center

Danny T. Y. Wu
1   Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, United States
2   Department of Pediatrics, University of Cincinnati, Cincinnati, Ohio, United States
,
Scott Vennemeyer
1   Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, United States
,
Kelly Brown
3   Heart Institute, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
,
Jason Revalee
4   DAAP School of Design, University of Cincinnati, Cincinnati, Ohio, United States
,
Paul Murdock
1   Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, United States
,
Sarah Salomone
1   Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, United States
,
Ashton France
3   Heart Institute, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
,
Katherine Clarke-Myers
3   Heart Institute, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
,
Samuel P. Hanke
2   Department of Pediatrics, University of Cincinnati, Cincinnati, Ohio, United States
3   Heart Institute, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
› Author Affiliations
Further Information

Publication History

19 April 2019

02 September 2019

Publication Date:
13 November 2019 (online)

Also available at
   Permissions and Reprints

Abstract

Background Interactive data visualization and dashboards can be an effective way to explore meaningful patterns in large clinical data sets and to inform quality improvement initiatives. However, these interactive dashboards may have usability issues that undermine their effectiveness. These usability issues can be attributed to mismatched mental models between the designers and the users. Unfortunately, very few evaluation studies in visual analytics have specifically examined such mismatches between these two groups.

Objectives We aimed to evaluate the usability of an interactive surgical dashboard and to seek opportunities for improvement. We also aimed to provide empirical evidence to demonstrate the mismatched mental models between the designers and the users of the dashboard.

Methods An interactive dashboard was developed in a large congenital heart center. This dashboard provides real-time, interactive access to clinical outcomes data for the surgical program. A mixed-method, two-phase study was conducted to collect user feedback. A group of designers (N = 3) and a purposeful sample of users (N = 12) were recruited. The qualitative data were analyzed thematically. The dashboards were compared using the System Usability Scale (SUS) and qualitative data.

Results The participating users gave an average SUS score of 82.9 on the new dashboard and 63.5 on the existing dashboard (p = 0.006). The participants achieved high task accuracy when using the new dashboard. The qualitative analysis revealed three opportunities for improvement. The data analysis and triangulation provided empirical evidence to the mismatched mental models.

Conclusion We conducted a mixed-method usability study on an interactive surgical dashboard and identified areas of improvements. Our study design can be an effective and efficient way to evaluate visual analytics systems in health care. We encourage researchers and practitioners to conduct user-centered evaluation and implement education plans to mitigate potential usability challenges and increase user satisfaction and adoption.

Keywords

data visualization - interface and usability - data quality - data analysis - dashboard - testing and evaluation - human–computer interaction - mixed (methodologies)

Protection of Human and Animal Subjects

The study protocol was reviewed by the University of Cincinnati Institutional Review Board (IRB) and determined as “non-human subject” research. All the research data were deidentified.


 

  • References

  • 1 Adler-Milstein J, Jha AK. HITECH Act drove large gains in hospital electronic health record adoption. Health Aff (Millwood) 2017; 36 (08) 1416-1422
    CrossrefPubMedGoogle Scholar
  • 2 Non-federal Acute Care Hospital Health IT Adoption and Use. Available at: https://dashboard.healthit.gov/apps/hospital-health-it-adoption.php . Accessed April 15, 2019
    PubMed
  • 3 Luo G. MLBCD: a machine learning tool for big clinical data. Health Inf Sci Syst 2015; 3: 3
    CrossrefPubMedGoogle Scholar
  • 4 Malykh VL, Rudetskiy SV. Approaches to medical decision-making based on big clinical data. J Healthc Eng 2018; 2018: 3917659
    PubMedGoogle Scholar
  • 5 Kahn MG, Eliason BB, Bathurst J. Quantifying clinical data quality using relative gold standards. AMIA Annu Symp Proc AMIA Symp 2010; 2010: 356-360
    PubMedGoogle Scholar
  • 6 Zhang Y, Sun W, Gutchell EM. , et al. QAIT: a quality assurance issue tracking tool to facilitate the improvement of clinical data quality. Comput Methods Programs Biomed 2013; 109 (01) 86-91
    CrossrefPubMedGoogle Scholar
  • 7 Caban JJ, Gotz D. Visual analytics in healthcare--opportunities and research challenges. J Am Med Inform Assoc 2015; 22 (02) 260-262
    CrossrefPubMedGoogle Scholar
  • 8 Gotz D, Borland D. Data-driven healthcare: challenges and opportunities for interactive visualization. IEEE Comput Graph Appl 2016; 36 (03) 90-96
    PubMedGoogle Scholar
  • 9 Bauer DT, Guerlain S, Brown PJ. The design and evaluation of a graphical display for laboratory data. J Am Med Inform Assoc 2010; 17 (04) 416-424
    CrossrefPubMedGoogle Scholar
  • 10 Plaisant C, Wu J, Hettinger AZ, Powsner S, Shneiderman B. Novel user interface design for medication reconciliation: an evaluation of Twinlist. J Am Med Inform Assoc 2015; 22 (02) 340-349
    CrossrefPubMedGoogle Scholar
  • 11 Mandalika VBH, Chernoglazov AI, Billinghurst M. , et al. A hybrid 2D/3D user interface for radiological diagnosis. J Digit Imaging 2018; 31 (01) 56-73
    CrossrefPubMedGoogle Scholar
  • 12 Quick Stats. These data visualizations of key data and statistics provide quick access to the latest facts and figures about health IT. Published February 6, 2019. Available at: https://dashboard.healthit.gov/quickstats/quickstats.php . Accessed April 15, 2019
    PubMedGoogle Scholar
  • 13 Huber TC, Krishnaraj A, Monaghan D, Gaskin CM. Developing an interactive data visualization tool to assess the impact of decision support on clinical operations. J Digit Imaging 2018; 31 (05) 640-645
    CrossrefPubMedGoogle Scholar
  • 14 Romero-Brufau S, Kostandy P, Maass KL. , et al. Development of data integration and visualization tools for the Department of Radiology to display operational and strategic metrics. AMIA Annu Symp Proc AMIA Symp 2018; 2018: 942-951
    PubMedGoogle Scholar
  • 15 Fletcher GS, Aaronson BA, White AA, Julka R. Effect of a real-time electronic dashboard on a rapid response system. J Med Syst 2017; 42 (01) 5
    PubMedGoogle Scholar
  • 16 Radhakrishnan K, Monsen KA, Bae S-H, Zhang W. ; Clinical Relevance for Quality Improvement. Visual analytics for pattern discovery in home care. Appl Clin Inform 2016; 7 (03) 711-730
    Article in Thieme ConnectPubMedGoogle Scholar
  • 17 Simpao AF, Ahumada LM, Larru Martinez B. , et al. Design and implementation of a visual analytics electronic antibiogram within an electronic health record system at a tertiary pediatric hospital. Appl Clin Inform 2018; 9 (01) 37-45
    Article in Thieme ConnectPubMedGoogle Scholar
  • 18 Hester G, Lang T, Madsen L, Tambyraja R, Zenker P. Timely data for targeted quality improvement interventions: use of a visual analytics dashboard for bronchiolitis. Appl Clin Inform 2019; 10 (01) 168-174
    Article in Thieme ConnectPubMedGoogle Scholar
  • 19 Karami M, Safdari R. From information management to information visualization: development of radiology dashboards. Appl Clin Inform 2016; 7 (02) 308-329
    Article in Thieme ConnectPubMedGoogle Scholar
  • 20 Federici S, Borsci S. Usability evaluation: models, methods, and applications. International encyclopedia of rehabilitation. Center for International rehabilitation research information and exchange (CIRRIE); 2010: 1-17
    Google Scholar
  • 21 Wu DTY, Chen AT, Manning JD. , et al. Evaluating visual analytics for health informatics applications: a systematic review from the American Medical Informatics Association Visual Analytics Working Group Task Force on Evaluation. J Am Med Inform Assoc 2019; 26 (04) 314-323
    CrossrefPubMedGoogle Scholar
  • 22 Norman DA. The Design of Everyday Things. Revised and expanded edition. New York, New York: Basic Books; 2013
    Google Scholar
  • 23 Mental Models. Available at: https://www.nngroup.com/articles/mental-models/ . Accessed April 15, 2019
    PubMed
  • 24 Kinzie MB, Cohn WF, Julian MF, Knaus WA. A user-centered model for web site design: needs assessment, user interface design, and rapid prototyping. J Am Med Inform Assoc 2002; 9 (04) 320-330
    CrossrefPubMedGoogle Scholar
  • 25 Kurzhals K, Hlawatsch M, Seeger C, Weiskopf D. Visual analytics for mobile eye tracking. IEEE Trans Vis Comput Graph 2017; 23 (01) 301-310
    CrossrefPubMedGoogle Scholar
  • 26 Ploderer B, Brown R, Seng LSD, Lazzarini PA, van Netten JJ. Promoting self-care of diabetic foot ulcers through a mobile phone app: user-centered design and evaluation. JMIR Diabetes 2018; 3 (04) e10105
    CrossrefPubMedGoogle Scholar
  • 27 Anoshin D, Rana H, Ma N. Mastering Business Intelligence with MicroStrategy Build World-Class Enterprise Business Intelligence Solutions with MicroStrategy 10. Birmingham, UK: Packt Publishing; 2016
    PubMedGoogle Scholar
  • 28 The Meaning Behind STAT Scores & Categories. Available at: https://www.luriechildrens.org/en/specialties-conditions/heart-center/volumes-outcomes/the-meaning-behind-stat-scores-categories/ . Accessed August 5, 2019
    PubMed
  • 29 van Lieshout R, Pisters MF, Vanwanseele B, de Bie RA, Wouters EJ, Stukstette MJ. Biofeedback in partial weight bearing: usability of two different devices from a patient's and physical therapist's perspective. PLoS One 2016; 11 (10) e0165199
    CrossrefPubMedGoogle Scholar
  • 30 Hamm J, Money AG, Atwal A, Ghinea G. Mobile three-dimensional visualisation technologies for clinician-led fall prevention assessments. Health Informatics J 2019; 25 (03) 788-810
    CrossrefPubMedGoogle Scholar
  • 31 Nitsch M, Adamcik T, Kuso S, Zeiler M, Waldherr K. Usability and engagement evaluation of an unguided online program for promoting a healthy lifestyle and reducing the risk for eating disorders and obesity in the school setting. Nutrients 2019; 11 (04) E713
    CrossrefPubMedGoogle Scholar
  • 32 Hirschmann J, Sedlmayr B, Zierk J. , et al. Evaluation of an interactive visualization tool for the interpretation of pediatric laboratory test results. Stud Health Technol Inform 2017; 243: 207-211
    PubMedGoogle Scholar
  • 33 He X, Zhang R, Rizvi R. , et al. ALOHA: developing an interactive graph-based visualization for dietary supplement knowledge graph through user-centered design. BMC Med Inform Decis Mak 2019; 19 (Suppl. 04) 150
    CrossrefPubMedGoogle Scholar
  • 34 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518
    Article in Thieme ConnectPubMedGoogle Scholar
  • 35 Brooke J. System Usability Scale (SUS): A Quick-And-Dirty Method of System Evaluation User Information. Read UK Digit Equip Co Ltd.; 1986: 43
    Google Scholar
  • 36 Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4 (03) 114-123
    PubMedGoogle Scholar
  • 37 Coe AM, Ueng W, Vargas JM. , et al. Usability testing of a web-based decision aid for breast cancer risk assessment among multi-ethnic women. AMIA Annu Symp Proc AMIA Symp 2016; 2016: 411-420
    PubMedGoogle Scholar
  • 38 Georgsson M, Staggers N. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. J Am Med Inform Assoc 2016; 23 (01) 5-11
    CrossrefPubMedGoogle Scholar
  • 39 Maguire M, Delahunt B. Doing a thematic analysis: a practical, step-by-step guide for learning and teaching scholars. AISHE-J Irel J Teach Learn High Educ 2017 ;9(3). Available at: http://ojs.aishe.org/index.php/aishe-j/article/view/335 . Accessed September 15, 2019
    PubMedGoogle Scholar
  • 40 Wenger MJ, Spyridakis JH. The relevance of reliability and validity to usability testing. IEEE Trans Prof Commun 1989; 32 (04) 265-271
    CrossrefPubMedGoogle Scholar
  • 41 Kessner M, Wood J, Dillon RF, West RL. On the reliability of usability testing. In: CHI '01 Extended Abstracts on Human Factors in Computing Systems - CHI '01. Seattle, Washington: ACM Press; 2001: 97
    Google Scholar