Download PDF
Appl Clin Inform 2024; 15(01): 164-169
DOI: 10.1055/a-2219-5175
Case Report

Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

Daniel J. Hekman
1   Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States
,
Hanna J. Barton
1   Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States
,
Apoorva P. Maru
1   Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States
,
Graham Wills
2   Department of Applied Data Science, UWHealth Hospitals and Clinics, Madison, Wisconsin, United States
,
Amy L. Cochran
3   Department of Population Health, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States
,
Corey Fritsch
2   Department of Applied Data Science, UWHealth Hospitals and Clinics, Madison, Wisconsin, United States
,
Douglas A. Wiegmann
4   Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin, United States
,
Frank Liao
2   Department of Applied Data Science, UWHealth Hospitals and Clinics, Madison, Wisconsin, United States
,
Brian W. Patterson
1   Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States
3   Department of Population Health, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States
4   Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin, United States
› Author Affiliations Funding This work is supported by the Agency for Healthcare Research and Quality grant number R18HS027735 (B.W.P., PI), U.S. Public Health Service, U.S. Department of Health and Human Services. This work is solely the output of the authors and does not represent the views of the Agency for Healthcare Research and Quality.
› Further Information
    Permissions and Reprints

Abstract

Background Existing monitoring of machine-learning-based clinical decision support (ML-CDS) is focused predominantly on the ML outputs and accuracy thereof. Improving patient care requires not only accurate algorithms but also systems of care that enable the output of these algorithms to drive specific actions by care teams, necessitating expanding their monitoring.

Objectives In this case report, we describe the creation of a dashboard that allows the intervention development team and operational stakeholders to govern and identify potential issues that may require corrective action by bridging the monitoring gap between model outputs and patient outcomes.

Methods We used an iterative development process to build a dashboard to monitor the performance of our intervention in the broader context of the care system.

Results Our investigation of best practices elsewhere, iterative design, and expert consultation led us to anchor our dashboard on alluvial charts and control charts. Both the development process and the dashboard itself illuminated areas to improve the broader intervention.

Conclusion We propose that monitoring ML-CDS algorithms with regular dashboards that allow both a context-level view of the system and a drilled down view of specific components is a critical part of implementing these algorithms to ensure that these tools function appropriately within the broader care system.

Keywords

dashboards - machine learning - clinical decision support - clinical informatics - statistical process control

Protection of Human and Animal Subjects

This larger intervention has been deemed minimal risk by our institutional review board and is registered at clinicaltrials.gov as NCT05810064; https://www.clinicaltrials.gov/study/NCT05810064.


Supplementary Material



Publication History

Received: 22 August 2023

Accepted: 28 November 2023

Accepted Manuscript online:
29 November 2023

Article published online:
28 February 2024

© 2024. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014; 33 (07) 1123-1131
    CrossrefPubMedGoogle Scholar
  • 2 Chen JH, Asch SM. Machine learning and prediction in medicine: beyond the peak of inflated expectations. N Engl J Med 2017; 376 (26) 2507-2509
    CrossrefPubMedGoogle Scholar
  • 3 Obermeyer Z, Emanuel EJ. Predicting the future: big data, machine learning, and clinical medicine. N Engl J Med 2016; 375 (13) 1216-1219
    CrossrefPubMedGoogle Scholar
  • 4 Brewer LC, Fortuna KL, Jones C. et al. Back to the future: achieving health equity through health informatics and digital health. JMIR Mhealth Uhealth 2020; 8 (01) e14512
    CrossrefPubMedGoogle Scholar
  • 5 Berlyand Y, Raja AS, Dorner SC. et al. How artificial intelligence could transform emergency department operations. Am J Emerg Med 2018; 36 (08) 1515-1517
    CrossrefPubMedGoogle Scholar
  • 6 Salwei ME, Carayon P, Hoonakker PLT. et al. Workflow integration analysis of a human factors-based clinical decision support in the emergency department. Appl Ergon 2021; 97: 103498
    CrossrefPubMedGoogle Scholar
  • 7 Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood) 2008; 27 (03) 759-769
    CrossrefPubMedGoogle Scholar
  • 8 Sikka R, Morath JM, Leape L. The quadruple aim: care, health, cost and meaning in work. BMJ Qual Saf 2015; 24 (10) 608-610
    CrossrefPubMedGoogle Scholar
  • 9 Nundy S, Cooper LA, Mate KS. The quintuple aim for health care improvement: a new imperative to advance health equity. JAMA 2022; 327 (06) 521-522
    CrossrefPubMedGoogle Scholar
  • 10 Carayon P, Wooldridge A, Hoonakker P, Hundt AS, Kelly MM. SEIPS 3.0: Human-centered design of the patient journey for patient safety. Appl Ergon 2020; 84: 103033
    CrossrefPubMedGoogle Scholar
  • 11 Jacobsohn GC, Leaf M, Liao F. et al. Collaborative design and implementation of a clinical decision support system for automated fall-risk identification and referrals in emergency departments. Healthc (Amst) 2022; 10 (01) 100598
    CrossrefPubMedGoogle Scholar
  • 12 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
    CrossrefPubMedGoogle Scholar
  • 13 Foster M, Albanese C, Chen Q. et al. Heart failure dashboard design and validation to improve care of veterans. Appl Clin Inform 2020; 11 (01) 153-159
    Article in Thieme ConnectPubMedGoogle Scholar
  • 14 Safranek CW, Feitzinger L, Joyner AKC. et al. Visualizing opioid-use variation in a pediatric perioperative dashboard. Appl Clin Inform 2022; 13 (02) 370-379
    Article in Thieme ConnectPubMedGoogle Scholar
  • 15 Nelson O, Sturgis B, Gilbert K. et al. A visual analytics dashboard to summarize serial anesthesia records in pediatric radiation treatment. Appl Clin Inform 2019; 10 (04) 563-569
    Article in Thieme ConnectPubMedGoogle Scholar
  • 16 Radhakrishnan K, Monsen KA, Bae SH, Zhang W. Clinical Relevance for Quality Improvement. Visual analytics for pattern discovery in home care. Appl Clin Inform 2016; 7 (03) 711-730
    Article in Thieme ConnectPubMedGoogle Scholar
  • 17 Singer SJ, Kellogg KC, Galper AB, Viola D. Enhancing the value to users of machine learning-based clinical decision support tools: A framework for iterative, collaborative development and implementation. Health Care Manage Rev 2022; 47 (02) E21-E31
    CrossrefPubMedGoogle Scholar
  • 18 Duckworth C, Chmiel FP, Burns DK. et al. Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19. Sci Rep 2021; 11 (01) 23017
    CrossrefPubMedGoogle Scholar
  • 19 Mišić VV, Rajaram K, Gabel E. A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission. NPJ Digit Med 2021; 4 (01) 98
    CrossrefPubMedGoogle Scholar
  • 20 Cieslak D, Chawla N. A framework for monitoring classifiers' performance: When and why failure occurs?. Knowl Inf Syst 2009; 18: 83-108
    CrossrefPubMedGoogle Scholar
  • 21 Bedoya AD, Economou-Zavlanos NJ, Goldstein BA. et al. A framework for the oversight and local deployment of safe and high-quality prediction models. J Am Med Inform Assoc 2022; 29 (09) 1631-1636
    CrossrefPubMedGoogle Scholar
  • 22 Feng J, Phillips RV, Malenica I. et al. Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. NPJ Digit Med 2022; 5 (01) 66
    CrossrefPubMedGoogle Scholar
  • 23 Patterson BW, Engstrom CJ, Sah V. et al. Training and interpreting machine learning algorithms to evaluate fall risk after emergency department visits. Med Care 2019; 57 (07) 560-566
    CrossrefPubMedGoogle Scholar
  • 24 Hekman D, Maru AP, Patterson BW. Dashboarding for Machine Learning-based Clinical Decision Support Implementation and Monitoring Toolkit. HIPxChange.org. 2023 . Accessed March 16, 2023 at: https://www.hipxchange.org/CDSDashboard
    PubMedGoogle Scholar
  • 25 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518
    Article in Thieme ConnectPubMedGoogle Scholar
  • 26 Jagannath S, Rifkin RM, Gasparetto CJ. et al; CONNECT MM Registry Investigators. Treatment journeys of patients with newly diagnosed multiple myeloma (NDMM): results from the Connect MM Registry. Clin Lymphoma Myeloma Leuk 2020; 20 (05) 272-276
    CrossrefPubMedGoogle Scholar
  • 27 Friendly M. Visions and re-visions of Charles Joseph Minard. J Educ Behav Stat 2002; 27 (01) 31-51
    CrossrefPubMedGoogle Scholar
  • 28 Slyngstad L. The contribution of variable control charts to quality improvement in healthcare: a literature review. J Healthc Leadersh 2021; 13: 221-230
    CrossrefPubMedGoogle Scholar
  • 29 Pimentel L, Barrueto Jr F. Statistical process control: separating signal from noise in emergency department operations. J Emerg Med 2015; 48 (05) 628-638
    CrossrefPubMedGoogle Scholar
  • 30 Berwick DM. Controlling variation in health care: a consultation from Walter Shewhart. Med Care 1991; 29 (12) 1212-1225
    CrossrefPubMedGoogle Scholar
  • 31 Thor J, Lundberg J, Ask J. et al. Application of statistical process control in healthcare improvement: systematic review. Qual Saf Health Care 2007; 16 (05) 387-399
    CrossrefPubMedGoogle Scholar
  • 32 Sotirovski D. Heuristics for iterative software development. IEEE Softw 2001; 18 (03) 66-73
    CrossrefPubMedGoogle Scholar
  • 33 Caldiera VRBG, Rombach HD. The goal question metric approach. Encycl Softw Eng 1994; 528-532
    PubMedGoogle Scholar
  • 34 R Core Team. R: A Language and Environment for Statistical Computing. 2020. Accessed December 13, 2023 at: https://www.R-project.org/
    PubMedGoogle Scholar
  • 35 Wickham H. ggplot2: Elegant Graphics for Data Analysis. 2016. Accessed December 13, 2023 at: https://ggplot2.tidyverse.org
    PubMedGoogle Scholar
  • 36 Allaire JJ, Xie Y, McPherson J. et al. rmarkdown: Dynamic Documents for R. 2021. Accessed December 13, 2023 at: https://github.com/rstudio/rmarkdown
    PubMedGoogle Scholar
  • 37 Xie Y, Dervieux C, Riederer E. R Markdown Cookbook. 2020. Accessed December 13, 2023 at: https://bookdown.org/yihui/rmarkdown-cookbook
    PubMedGoogle Scholar
  • 38 Wickham H, Averick M, Bryan J. et al. Welcome to the tidyverse. J Open Source Softw 2019; 4 (43) 1686
    CrossrefPubMedGoogle Scholar
  • 39 Wilkinson L. The Grammar of Graphics. 2nd ed. New York, NY:: Springer;; 2005
    Google Scholar
  • 40 Fairbanks RJ, Bisantz AM. Understanding better how clinicians work. Ann Emerg Med 2011; 58 (02) 123-125
    CrossrefPubMedGoogle Scholar
  • 41 Donabedian A. The quality of care. How can it be assessed?. JAMA 1988; 260 (12) 1743-1748
    CrossrefPubMedGoogle Scholar
  • 42 Carayon P, Schoofs Hundt A, Karsh BT. et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 (Suppl. 01) i50-i58
    CrossrefPubMedGoogle Scholar
  • 43 Noyez L. Control charts, Cusum techniques and funnel plots. A review of methods for monitoring performance in healthcare. Interact Cardiovasc Thorac Surg 2009; 9 (03) 494-499
    CrossrefPubMedGoogle Scholar
  • 44 Hekman DJ, Cochran AL, Maru AP. et al. Effectiveness of an emergency department-based machine learning clinical decision support tool to prevent outpatient falls among older adults: protocol for a quasi-experimental study. JMIR Res Protoc 2023; 12: e48128
    CrossrefPubMedGoogle Scholar
  • 45 Juran JM. Quality-control handbook. New York, NY: McGraw-Hill; 1951. . Available at: https://cir.nii.ac.jp/crid/1130000795132214656
    Google Scholar