Download PDF
Appl Clin Inform 2019; 10(05): 859-869
DOI: 10.1055/s-0039-1698466
Special Topic: Visual Analytics in Healthcare
Georg Thieme Verlag KG Stuttgart · New York

Usability Testing of an Interactive Dashboard for Surgical Quality Improvement in a Large Congenital Heart Center

Danny T. Y. Wu
1   Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, United States
2   Department of Pediatrics, University of Cincinnati, Cincinnati, Ohio, United States
,
Scott Vennemeyer
1   Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, United States
,
Kelly Brown
3   Heart Institute, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
,
Jason Revalee
4   DAAP School of Design, University of Cincinnati, Cincinnati, Ohio, United States
,
Paul Murdock
1   Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, United States
,
Sarah Salomone
1   Department of Biomedical Informatics, University of Cincinnati, Cincinnati, Ohio, United States
,
Ashton France
3   Heart Institute, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
,
Katherine Clarke-Myers
3   Heart Institute, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
,
Samuel P. Hanke
2   Department of Pediatrics, University of Cincinnati, Cincinnati, Ohio, United States
3   Heart Institute, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, United States
› Author Affiliations
Further Information

Publication History

19 April 2019

02 September 2019

Publication Date:
13 November 2019 (online)

Also available at
   Permissions and Reprints
Zoom Image

Abstract

Background Interactive data visualization and dashboards can be an effective way to explore meaningful patterns in large clinical data sets and to inform quality improvement initiatives. However, these interactive dashboards may have usability issues that undermine their effectiveness. These usability issues can be attributed to mismatched mental models between the designers and the users. Unfortunately, very few evaluation studies in visual analytics have specifically examined such mismatches between these two groups.

Objectives We aimed to evaluate the usability of an interactive surgical dashboard and to seek opportunities for improvement. We also aimed to provide empirical evidence to demonstrate the mismatched mental models between the designers and the users of the dashboard.

Methods An interactive dashboard was developed in a large congenital heart center. This dashboard provides real-time, interactive access to clinical outcomes data for the surgical program. A mixed-method, two-phase study was conducted to collect user feedback. A group of designers (N = 3) and a purposeful sample of users (N = 12) were recruited. The qualitative data were analyzed thematically. The dashboards were compared using the System Usability Scale (SUS) and qualitative data.

Results The participating users gave an average SUS score of 82.9 on the new dashboard and 63.5 on the existing dashboard (p = 0.006). The participants achieved high task accuracy when using the new dashboard. The qualitative analysis revealed three opportunities for improvement. The data analysis and triangulation provided empirical evidence to the mismatched mental models.

Conclusion We conducted a mixed-method usability study on an interactive surgical dashboard and identified areas of improvements. Our study design can be an effective and efficient way to evaluate visual analytics systems in health care. We encourage researchers and practitioners to conduct user-centered evaluation and implement education plans to mitigate potential usability challenges and increase user satisfaction and adoption.

Keywords

data visualization - interface and usability - data quality - data analysis - dashboard - testing and evaluation - human–computer interaction - mixed (methodologies)

Protection of Human and Animal Subjects

The study protocol was reviewed by the University of Cincinnati Institutional Review Board (IRB) and determined as “non-human subject” research. All the research data were deidentified.