COVID-19 and the Electronic Health Record: Tool Design and Evolution at the U.S. Pandemic Epicenter

Abstract

Objective We detail inpatient electronic health record (EHR) system tools created at Mount Sinai Health System for the clinical management of patients with coronavirus disease 2019 (COVID-19) during the early pandemic months in the U.S. epicenter, New York City. We discuss how we revised these tools to create a robust Care pathway, unlike other tools reported, that helped providers care for these patients as guidelines evolved.

Methods Mount Sinai Health System launched a Command Center on March 8, 2020. The Chief Medical Information Officer launched a workgroup of clinical informaticists and Epic analysts tasked with rapidly creating COVID-19-related EHR tools for the inpatient setting.

Results Initial EHR tools focused on inpatient order sets for care standardization and resource utilization. In preparation for a fall 2020-winter 2021 surge, we created a clinician-facing, integrated Care pathway incorporating additional Epic System-specific tools: a Care Path, a dedicated Navigator, Summary and Timeline Reports, and SmartTexts.

Discussion Initial tools offered standard functionality but included complex decision-making support to account for the lack of COVID-19 clinical knowledge, operational challenges during a dramatic patient surge, and resource limitations. We revised content and built a more comprehensive Care pathway that provided real-time clinical data along with treatment recommendations as knowledge evolved, e.g., convalescent plasma.

Conclusion We have provided a framework that can inform future informaticists in developing EHR tools during an evolving pandemic.

Keywords

inpatient - order entry - specific types - clinical information systems - data visualization - general medicine admission - inpatient care - environment - clinical care

Introduction

New York City was the earliest U.S. epicenter of the coronavirus disease 2019 (COVID-19) pandemic, with its first case in March 2020 and a surge to more than 5,000 confirmed cases per day.[1] [Fig. 1] provides context on the initial surge. Others have previously described organizational structures,[1] documentation and order panels,[2] [3] data visualization tools,[3] [4] reporting tools,[3] [5] [6] clinical decision support (CDS),[7] [8] and operational informatics tools.[9] [10]

Fig. 1 For context, daily cases in New York City (in blue) and number of COVID-19 hospitalized patients at MSHS (in red) are presented here.[1] [2] Rapidly increasing COVID-19 related admissions in Spring 2020 during the first surge of the pandemic in the United States necessitated expedited development of EHR tools in the face of many unknown factors. EHR, electronic health record; MSHS, Mount Sinai Health System.

Care pathways have been described in the informatics literature as “structured multidisciplinary care plans that detail essential steps in the care of patients with a specific clinical problem [and] offer a structured means of developing and implementing local protocols of care based on clinical guidelines.”[8] Care pathways, particularly those integrated into the electronic health record (EHR), have been shown to improve in-hospital outcomes,[11] [12] increase the detection, diagnosis, and treatment of specified conditions,[13] and improve utilization of resources.[14] [15] [16] Care pathways represent the evolution of previously described tools that integrated clinical guidelines or care recommendations within the provider workflow.[17] This report outlines the development of order sets, novel data visualization tools, and a novel COVID-19 Care pathway within the context of a rapidly escalating pandemic.

Objective

In this case report, we detail the inpatient EHR clinical tools created at the Mount Sinai Health System (MSHS) for the management of patients with COVID-19 during the early months of the pandemic. We describe how these tools evolved into a novel clinician-facing, integrated Care pathway that aggregates clinical information and COVID-19-specific tools into a navigator that interprets the patient's clinical status and provides treatment recommendations and lessons we learned about pandemic realities versus best practices in the literature.

Methods

The Mount Sinai Health System is an academic medical center and health system comprising more than 300 hundred ambulatory practices and eight hospitals in the New York City metropolitan area, six of which currently use the Epic EHR (Epic Systems Corporation, Verona, Wisconsin, United States). MSHS launched a pandemic Command Center (CC) on March 8, 2020, composed of senior leadership across the health system, including the president and chief executive officers of each hospital, the system's chief medical information officer (CMIO), and system chairs for Infectious Disease, Hospital Epidemiology, and Pulmonary/Critical Care.

The CMIO formed a small informatics workgroup (three clinician informaticists and 6–8 Epic analysts) tasked with creating as quickly as possible COVID-19-related EHR tools for the inpatient setting. All requests came from the CC, and all tools developed were reviewed by the appropriate health system clinical department chairs prior to implementation. The workgroup created clinical decision support, computerized provider order entry, and data visualization tools deemed essential for caring for COVID-19 patients. The workgroup presented bi-weekly in meetings with a standing EHR-governance council of about two dozen inpatient clinical informaticists from across the system who provided critical feedback during tool development. [Fig. 1] highlights the daily case rate in New York City, hospitalization rates within our institution, and provides context against which this build work was occurring.[18] For reporting purposes here, the efforts of this workgroup are divided into two phases, Phase 1 (mid-March to mid-June 2020) and Phase 2 (September to December 2020). The timeline of the efforts of this workgroup is presented in [Fig. 2] and accompanies [Table 1], which outlines major milestones for each tool.

Fig. 2 COVID-19 EHR tools development timeline at MSHS. EHR, electronic health record; MSHS, Mount Sinai Health System.

Table 1 COVID-19 EHR tools development milestones at MSHS

EHR tool type	Date	Milestone description	Key update aspects
COVID-19 order set	March 20, 2020	Launched in production
	May 12, 2020	Update 1	Added logic to show/hide certain items based on whether the patient is COVID+
	October 07, 2020	Update 2	Content optimization[a]
	November 17, 2020	Update 3	Added order to place patient on COVID-19 pathway (care path)
Admission order set SmartGroup	March 24, 2020	Launched in production	[a]
	May 12, 2020	Update 1	Added logic to show/hide certain items based on whether the patient is COVID+
	October 07, 2020	Update 2	[a]
	November 17, 2020	Update 3	Added order to place patient on COVID-19 pathway (care path)
ICU admission order set SmartGroup	March 24, 2020	Launched in production	[a]
	May 12, 2020	Update 1	Added logic to show/hide certain items based on whether the patient is COVID+
	October 07, 2020	Update 2	[a]
Convalescent plasma order set	March 27, 2020	Launched in production	[a]
	October 21, 2020	Update 1	Included restrictions based on required laboratory results and documentation
	January 11, 2021	Update 2	Updated restrictions to account for transfers between hospitals
	February 04, 2021	Update 3	Displayed criteria met vs. not met to end users, saving troubleshooting time for clinicians
Summary report	March 27, 2020	Launched in production	[a]
	October 21, 2020	Update 1	Added to new COVID-19 navigator
	November 17, 2020	Update 2	Added link to new COVID-19 care path, COVID-19 timeline report
	December 09, 2020	Update 3	[a]
Anticoagulation order panel	April 10, 2020	Launched in production	[a]
	April 28, 2020	Update 1	[a]
Navigator	October 21, 2020	Launched in production	[a]
	November 17, 2020	Navigator update 1	Added treatment guidance section
	December 09, 2020	Navigator update 2	[a]
Care path	November 17, 2020	Launched in production	[a]
	December 09, 2020	Care path update 1	Update anticoagulation treatment guidance

^a Content optimization: e.g., addition or removal of laboratory tests or updates to medication dosages that reflected changes in the institution's treatment guidelines.

Results

Phase 1 Inpatient Tools

There were many unknowns regarding caring for patients with COVID-19 in the inpatient setting during the early phase of EHR tool building. Our informatics workgroup aimed to quickly and efficiently create the inpatient tools needed to disseminate information and standardize care for patients with COVID-19.

The first comprehensive inpatient tool created was the COVID-19 order set. With prior experience in bundling diagnostic and treatment options within one order set (for conditions such as community-acquired pneumonia), this seemed like the natural first step in providing decision support for clinicians. The order set included testing for SARS-CoV-2, appropriate isolation precautions, and preselected laboratory tests. Providers would select clinical severity, then were provided with corresponding medication recommendations, including post-treatment monitoring. A set of cascading questions regarding clinical severity, weight, and creatinine clearance was later added to guide anticoagulation dosing ([Fig. 3]). Needs were identified by clinical leadership in response to: updated treatment recommendations; availability of resources (including testing capacity, medication supply, etc.); and unforeseen challenges that required course redirection.

Fig. 3 Screenshot of the COVID-19 order set. Upon clinician selection of a particular severity of disease, the order set provides corresponding medication options. Treatment recommendations were incorporated within the order set. Because changes to recommendations occurred rapidly and frequently as COVID-19 knowledge grew and the complexity of those recommendations, cascading orders were used to ensure appropriate dosing for anti-coagulation.

As COVID-19 hospitalization rates skyrocketed, and non-COVID related admissions plummeted, these standalone order sets were integrated into the general medicine admission and intensive care admission order sets to streamline the ordering process. A patient-specific summary report was created to centralize important clinical information including vital signs, oxygen requirement or ventilator settings, inflammatory markers over the last 72 hours, microbiology data, and details about the treatment team.

Phase 2 Inpatient Tools

As COVID-19 cases decreased over the summer, MSHS began preparations to manage a potential fall/winter surge. We created an EHR-integrated clinical Care pathway that provided decision support and treatment guidance and included the following Epic-specific tools: a Care Path, a dedicated Navigator, Order Sets, Summary and Timeline Reports, and SmartTexts.

Care Path

A Care Path is an Epic-specific decision support tool that visually outlines treatment options for a particular condition and updates in real-time based on the documentation in the chart. Our prior experience with Care Paths in the inpatient setting involved management of patients with respiratory failure in the intensive care unit. The benefits of this tool include allowing providers to easily track clinical progress and suggest appropriate interventions at various stages of the patient's clinical course.

In designing a COVID-19 care path ([Fig. 4]), oxygen supplementation became the designated milestone to track across the patient's clinical course because disease severity was defined by the degree of respiratory failure. All available oxygen delivery devices were associated with one of the milestones shown. Milestones were “achieved” based on clinical data documented in nursing vitals flowsheets and updated in real time. Each milestone had a corresponding treatment recommendation, including steroid and anti-coagulation dosing. The Care Path also differentiated between escalating and de-escalating oxygen requirements, and provided discharge recommendations when oxygen requirements decreased below 3 L of oxygen by nasal cannula. The Care Path was triggered by a specific Care Path order in the COVID-19 order set; providers received an alert recommending the Care Path if a patient tested positive for SARS-CoV-2 and did not have an active COVID-19 Care Path order. An overwhelming majority of patients admitted with COVID-19 were placed on the pathway since its launch ([Table 2]).

Fig. 4 Pictured is the Clinical Care Path, which clinicians access from the Summary report within the COVID-19 navigator. The Care Path is organized by oxygen delivery device, with corresponding treatment recommendations. Patients proceed along the Care Path based on nursing flowsheet documentation of oxygen supplementation. Each treatment option is associated with a corresponding medication or nursing order. For example, an order for dexamethasone (ordered while the patient is documented to be on nasal cannula) will be displayed as a completed step. Completed steps are time stamped. Dark blue boxes with an icon of a person represent the most recent oxygen requirement or therapeutic intervention.

Table 2 COVID-19 care pathway utilization in the EHR at MSHS

	Admitted patients, COVID +	Patients placed on pathway	%
Nov 2020	214	194	90.65%
Dec 2020	966	808	83.64%
Jan 2021	1054	977	92.69%
Feb 2021	895	857	95.75%

Navigator

Based on our earlier experience with a COVID-19 Summary Report, we believed it was important to create a centralized location for the management of patients with COVID-19. In Epic, a dedicated Navigator allowed for the various tools we developed to be accessed from the same screen. Importantly, we were able to dynamically display the Navigator more prominently in the patient's chart based on infection status (i.e., person under investigation or positive SARS-CoV-2 PCR result). This addressed a shortcoming of the COVID-19 summary report, which was the inability to automatically display the report upon opening the chart of a COVID-19 patient.

We leveraged the existing COVID-19 summary report in the COVID-19 navigator and made optimizations ([Fig. 5]), including a direct link to the COVID-19 Care Path. SmartText was added to the top of the summary report highlighting treatment recommendations based on the patient's documented oxygen therapy. This SmartText directly corresponded with the COVID-19 Care Path.

Fig. 5 COVID-19 summary report within the navigator. The Summary Report includes hyperlinks to the MSHS Intranet site which includes expanded treatment recommendations and an ongoing list of COVID-19-related trials. Similar to the Care Path, nursing documentation of the patient's oxygen requirement triggers a corresponding SmartText that displays treatment recommendations. This SmartText is updated in real time (e.g., when a patient is de-escalated from non-rebreather to 2L nasal cannula, the SmartText will automatically refresh with the corresponding treatment recommendations based on oxygen de-escalation). The Summary Report directly links to the Care Path and Timeline Report. Presented with permission, © 2021 Epic Systems Corporation.

We also provided a link to a custom-built COVID-19 timeline report which displays COVID-19-specific treatments chronologically along with oxygen requirements or ventilator settings, use of prone positioning, and graphical representation of inflammatory markers and fever curves ([Fig. 6]). This timeline view provides another data visualization tool by which the provider can trace the patient's clinical course during the hospitalization.

Fig. 6 The Timeline Report allows for a comprehensive overview of the patient's treatment course during the entire hospitalization. Changes in treatment (e.g., an increase in the dose of anti-coagulation or administration of convalescent plasma) can be tracked alongside changes in oxygen delivery, temperature, and inflammatory markers. Presented with permission, © 2021 Epic Systems Corporation.

Convalescent plasma was available under the guidance of the Federal Drug Administration's Emergency Use Authorization, which necessitated strict documentation requirements. To ensure adherence, an order set and consent documentation was created for convalescent plasma and made accessible within the COVID-19 navigator. Order Set restrictors with multiple rules were created to check if result and documentation criteria were met; if one of the criteria was not met, the convalescent plasma order would not be available to the provider and would instead display criteria needed for ordering convalescent plasma.

Discussion

Although other institutions described inpatient clinical order sets and CDS tools meant to standardize and simplify care for patients with COVID-19, we were in the unique position of creating EHR-based tools during the first and then-unprecedented surge, and continued to improve our tools for use during the ongoing pandemic. We applied known best practices and learned pragmatic lessons for building EHR-based tools that may help health systems in future pandemics. Our approach may help others move more rapidly to develop more sophisticated tools to aid clinicians during similar crises.

First, during that spring surge, speed and simplicity were keys to creating successful tools. Our early efforts were focused on straightforward tools that could be built quickly and updated relatively easily. CDS ease-of-use, always important,[19] was particularly critical because many providers with little inpatient experience were being deployed to inpatient care. Though technically simple, designing these tools required complicated clinical decision-making. Potential diagnostic and treatment options were being announced at a rapid rate, often outside the standard channel of peer-reviewed academic journals. This clinical uncertainty added a degree of caution to what options we made available to providers. Moreover, order set composition had to balance clinical necessity (e.g., trending inflammatory markers), and resource limitations (e.g., the turnaround time for obtaining a laboratory result).

Second, employing best practices to create well-integrated CDS tools made treatment recommendations more readily available to providers within the time constraints of an evolving pandemic.[19] Integrating treatment recommendations within the order set was an essential component to this. Additionally, noting that our initial data visualization tools were not centralized within the EHR, potentially discouraging use, utilization of the navigator aggregated these tools, thereby embedding them within the chart review workflow next to these clinical decision support tools. We chose not to incorporate intrusive Best Practice Advisory pop-up alerts to avoid alert fatigue.[20] The tools also decreased the need to leave the EHR, which would have been the case if we relied on email communication or an intranet site.

Limitations to our clinical tools and our processes offer a learning opportunity. There are limits to standardizing care within the EHR, regardless of EHR system used and the specific functionalities it offers. Our tools relied on clinician judgment, rather than forced functions, to adhere to treatment recommendations (excluding the convalescent plasma workflow). In practice that meant that although our order set included recommended dosing for steroids, providers were still able to order these medications outside of the order set at a dosage of their choosing. Given previously discussed challenges early on, particularly the limited availability of evidence to guide therapy, we sacrificed some forced standardization to allow providers the flexibility to consider recommendations within the context of the patients for which they were caring.

Additionally, the complexity of managing patients with COVID-19 makes it difficult to create a linear or easy to digest treatment algorithm. Though respiratory status is the most important indicator, our tool does not provide guidance on how to manage COVID-19 related complications such as delirium, thromboembolism, or renal failure. These trade-offs were necessary in the hopes of creating a tool that could be used by all providers, particularly if we again had to rely on those re-deployed from other specialties. Similar trade-offs should be recognized and accepted in a similar future pandemic.

An additional limitation was our inability to track utilization of all the tools we created. Our primary goal was to provide clinicians with recommendations and tools needed to care for patients with COVID-19, and to ensure that resources were utilized judiciously. Quality or outcome measures were not identified because there was no framework to establish reasonable benchmarks. We therefore focused reporting resources on other measures (e.g., ventilator utilization). Additionally, though we were able to ensure that a large majority of patients were placed on the Care Path, we did not have access to information such as time spent within the summary report or various other activities within the navigator. This meant that we did not have sufficient data to further optimize our tools, such as improving tools providers used the most or removing those they did not use.

Lastly, our experience highlights the limitations of being able to apply previously described frameworks during a pandemic. There is previous literature on user-centered or human-centered design and their effectiveness in creating useful EHR tools.[21] [22] Our EHR governance council included multiple board-certified clinical informaticists and a PhD-prepared informaticist, all expert members familiar with the CDS, usability, and user-centered design literature, who have previously employed these literature-driven best-practices for EHR build, usability testing, implementation, and user-centered optimization.[23] [24] [25] Unfortunately, the first surge involved innumerable unknowns, making it difficult to proceed in a deliberative, more structured manner. Shiffman et al have previously described validated frameworks to implement guidelines, yet our tools were created in an environment where guidelines did not exist, and recommendations were ever-changing.[26] Moreover, validated design processes, such as Schiffman's, are time and labor intensive and not practical when all personnel were in crisis-response mode with extended enterprise-designated pandemic duties. Jakob Nielsen's simplified model of usability engineering estimated that parts of the process would take at least 105 hours to complete.[27] This would have been untenable in our situation where decisions such as changes to anti-coagulation recommendations were made and released within 48 hours. Lastly, rapidly changing events precluded structured processes for evaluating and updating tools. This is not surprising given that established frameworks to evaluate deployed EHR tools are known to be complex schemes with “multiple dimensions, categories, and measures that can be difficult to understand and apply in practice.”[28] Even a simplified, practically focused framework such as SEIPS 101 may be too consuming to apply in another emergent pandemic situation.[29] Future research should focus on ultrarapid processes to develop and evaluate EHR tools. However, a key takeaway is that health systems that have access to a corps of literature- and best practice-informed clinical informatics experts[7] or integrate them into standing EHR-governance processes,[30] as we did, will be best positioned to act decisively in crises when more rigorous, intensive feedback/evaluation processes are impossible.

Conclusion

We detailed inpatient EHR clinical tools we developed for the management of COVID-19 patients, how we have refined these tools since the initial New York City surge, and how the tools became a robust clinician-facing, integrated Care pathway to provide comprehensive care to patients with COVID-19. Our best-practices approach in EHR-tool development to meet immediate clinical needs, then leveraging additional functionalities and evolving knowledge, and lessons learned, may inform and aid future informatics and EHR-build pandemic responses.

Conflict of Interest

None declared.

• References

• 1 Grange ES, Neil EJ, Stoffel M. et al. Responding to COVID-19: the UW medicine information technology services experience. Appl Clin Inform 2020; 11 (02) 265-275
• 2 Deeds SA, Hagan SL, Geyer JR. et al. Leveraging an electronic health record note template to standardize screening and testing for COVID-19. Healthc (Amst) 2020; 8 (03) 100454
• 3 Reeves JJ, Hollandsworth HM, Torriani FJ. et al. Rapid response to COVID-19: health informatics support for outbreak management in an academic health system. J Am Med Inform Assoc 2020; 27 (06) 853-859
• 4 Dixon BE, Grannis SJ, McAndrews C. et al. Leveraging data visualization and a statewide health information exchange to support COVID-19 surveillance and response: application of public health informatics. J Am Med Inform Assoc 2021; 28 (07) 1363-1373
• 5 Salway RJ, Silvestri D, Wei EK, Bouton M. Using information technology to improve COVID-19 care at New York City Health + Hospitals. Health Aff (Millwood) 2020; 39 (09) 1601-1604
• 6 Fareed N, Swoboda CM, Chen S, Potter E, Wu DTY, Sieck CJUS. U.S. COVID-19 state government public dashboards: an expert review. Appl Clin Inform 2021; 12 (02) 208-221
• 7 Lin C-T, Bookman K, Sieja A. et al. Clinical informatics accelerates health system adaptation to the COVID-19 pandemic: examples from Colorado. J Am Med Inform Assoc 2020; 27 (12) 1955-1963
• 8 Davis MW, McManus D, Koff A. et al. Repurposing antimicrobial stewardship tools in the electronic medical record for the management of COVID-19 patients. Infect Control Hosp Epidemiol 2020; 41 (11) 1335-1337
• 9 Blazey-Martin D, Barnhart E, Gillis Jr J, Vazquez GA. Primary care population management for COVID-19 patients. J Gen Intern Med 2020; 35 (10) 3077-3080
• 10 Knighton AJ, Ranade-Kharkar P, Brunisholz KD. et al. Rapid implementation of a complex, multimodal technology response to COVID-19 at an integrated community-based health care system. Appl Clin Inform 2020; 11 (05) 825-838
• 11 Edholm K, Lappé K, Kukhareva P. et al. Reducing diabetic ketoacidosis intensive care unit admissions through an electronic health record-driven, standardized care pathway. J Healthc Qual 2020; 42 (05) e66-e74
• 12 Aeyels D, Bruyneel L, Sinnaeve PR. et al. Care pathway effect on in-hospital care for ST-elevation myocardial infarction. Cardiology 2018; 140 (03) 163-174
• 13 Breton J, Witmer CM, Zhang Y. et al. Utilization of an electronic medical record-integrated dashboard improves identification and treatment of anemia and iron deficiency in pediatric inflammatory bowel disease. Inflamm Bowel Dis 2021; 27 (09) 1409-1417
• 14 Campbell H, Hotchkiss R, Bradshaw N, Porteous M. Integrated care pathways. BMJ 1998; 316 (7125): 133-137
• 15 Dhaliwal JS, Goss F, Whittington MD. et al. Reduced admission rates and resource utilization for chest pain patients using an electronic health record-embedded clinical pathway in the emergency department. J Am Coll Emerg Physicians Open 2020; 1 (06) 1602-1613
• 16 Sicotte C, Lapointe J, Clavel S, Fortin MA. Benefits of improving processes in cancer care with a care pathway-based electronic medical record. Pract Radiat Oncol 2016; 6 (01) 26-33
• 17 Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 1999; 6 (02) 104-114
• 18 NYC Coronavirus Disease. 2019 COVID-19 Data. Accessed January 8, 2021 at: https://github.com/nychealth/coronavirus-data
• 19 Sirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA. Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives. J Healthc Inf Manag 2009; 23 (04) 38-45
• 20 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14 (04) 415-423
• 21 Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform 2005; 38 (01) 75-87
• 22 Brunner J, Chuang E, Goldzweig C, Cain CL, Sugar C, Yano EM. User-centered design to improve clinical decision support in primary care. Int J Med Inform 2017; 104: 56-64
• 23 Genes N, Kim MS, Thum FL. et al. Usability evaluation of a clinical decision support system for geriatric ED pain treatment. Appl Clin Inform 2016; 7 (01) 128-142
• 24 Otokiti AU, Craven CK, Shetreat-Klein A, Cohen S, Darrow B. Beyond getting rid of stupid stuff in the electronic health record (Beyond-GROSS): protocol for a user-centered, mixed-method intervention to improve the electronic health record system. JMIR Res Protoc 2021; 10 (03) e25148
• 25 McNeely J, Adam A, Rotrosen J. et al. Comparison of methods for alcohol and drug screening in primary care clinics. JAMA Netw Open 2021; 4 (05) e2110721
• 26 Shiffman RN, Michel G, Essaihi A, Thornquist E. Bridging the guideline implementation gap: a systematic, document-centered approach to guideline implementation. J Am Med Inform Assoc 2004; 11 (05) 418-426
• 27 Nielsen J. Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier. Cost-Justifying Usability. Boston, MA: Academic Press, Inc.; 1994: 245-272
• 28 Craven CK, Doebbeling B, Furniss D, Holden RJ, Lau F, Novak LL. Evidence-based health informatics frameworks for applied use. Stud Health Technol Inform 2016; 222: 77-89
• 29 Holden RJ, Carayon P. SEIPS 101 and seven simple SEIPS tools. BMJ Qual Saf 2021; 30 (11) 901-910
• 30 Wright A, Sittig DF, Ash JS. et al. Governance for clinical decision support: case studies and recommended practices from leading institutions. J Am Med Inform Assoc 2011; 18 (02) 187-194

Publication History

Received: 09 July 2021

Accepted: 19 July 2022

Article published online:
18 September 2022