Graphical Presentations of Clinical Data in a Learning Electronic Medical Record

Abstract

Background Complex electronic medical records (EMRs) presenting large amounts of data create risks of cognitive overload. We are designing a Learning EMR (LEMR) system that utilizes models of intensive care unit (ICU) physicians' data access patterns to identify and then highlight the most relevant data for each patient.

Objectives We used insights from literature and feedback from potential users to inform the design of an EMR display capable of highlighting relevant information.

Methods We used a review of relevant literature to guide the design of preliminary paper prototypes of the LEMR user interface. We observed five ICU physicians using their current EMR systems in preparation for morning rounds. Participants were interviewed and asked to explain their interactions and challenges with the EMR systems. Findings informed the revision of our prototypes. Finally, we conducted a focus group with five ICU physicians to elicit feedback on our designs and to generate ideas for our final prototypes using participatory design methods.

Results Participating physicians expressed support for the LEMR system. Identified design requirements included the display of data essential for every patient together with diagnosis-specific data and new or significantly changed information. Respondents expressed preferences for fishbones to organize labs, mouseovers to access additional details, and unobtrusive alerts minimizing color-coding. To address the concern about possible physician overreliance on highlighting, participants suggested that non-highlighted data should remain accessible. Study findings led to revised prototypes, which will inform the development of a functional user interface.

Conclusion In the feedback we received, physicians supported pursuing the concept of a LEMR system. By introducing novel ways to support physicians' cognitive abilities, such a system has the potential to enhance physician EMR use and lead to better patient outcomes. Future plans include laboratory studies of both the utility of the proposed designs on decision-making, and the possible impact of any automation bias.

Keywords

electronic health records - intensive care units - data display - user-computer interface - software design - cognition

Background and Significance

The volume of clinical information collected by modern electronic medical record (EMR) systems creates a potential for cognitive overload, especially in data-rich environments such as intensive care units (ICUs).[1] To draw physicians' attention to high-value information, we are designing a Learning EMR (LEMR) system that utilizes statistical models of ICU physicians' data access patterns to identify data that are most likely to be sought for each patient.[2] [3] [4] To be successful, this system must combine accurate predictions of data seeking behavior with information displays that meet physicians' needs. This paper presents the results of a qualitative investigation of the design of a LEMR system display.

Cognitive Overload in Intensive Care Units: the Performance-Altering Effects of Electronic Medical Record Systems

Physicians must allocate limited cognitive resources to multiple competing tasks.[2] EMR systems should provide excellent cognitive support, but often fall short of doing so.[5] [6] [7] [8] Most commercial systems have limited capabilities for visualizing complex data,[9] and fragmentation of information across multiple screens increases the difficulty of collating data and identifying trends and anomalies.[10] Reductions in efficiency due to overabundant data and suboptimal displays may impact physicians' decision-making and performance,[11] [12] potentially leading to medical errors and patient harm.[9] [13] [14] [15] [16] EMR systems that provide cognitive support may help improve patient outcomes.[17]

Approaches to Enhancing Cognitive Performance in the Intensive Care Unit

Despite several studies investigating ICU physicians' data needs and information-seeking processes,[18] [19] [20] [21] [22] a thorough understanding of the cognitive support needs of ICU clinicians is yet to be achieved.[18] [21] Proposed designs intended to provide cognitive support to physicians[23] have explored techniques including encoding health information using visual attributes such as color, position, size, or shape.[24] [25] [26] Displays have also used visualizations to summarize patient data and medical histories,[21] [25] [26] [27] [28] [29] and to aggregate information from different sources in summary views,[10] [23] [30] [31] organized by organ system[30] or around clinical concepts.[24] [30] [32] Other efforts have explored visual highlighting of trends[22] and key data in EMRs[33] or clinical notes,[34] and the adoption of configurable user interfaces.[35] [36] While many of these approaches have been shown to reduce cognitive load[2] [21] [23] [37] and improve physician decision-making,[30] [32] [38] they are not commonly used in healthcare.[7]

A Data-Driven Learning Electronic Medical Record System

We propose a novel data-driven approach that has the potential to enhance ICU physicians' performance. The LEMR system uses data access patterns from past patient cases to train models capable of predicting which data items are important in understanding a patient's condition and how to treat it.[2] Items predicted to be of high value are highlighted in the LEMR system's user interface, helping physicians focus on essential data.[2] Preliminary prediction models having AUROC values as high as 0.92[2] [4] support that the data-driven approach is promising.

Objectives

The LEMR system requires a display capable of effectively highlighting the most relevant information for each patient. To better understand user needs[39] and to create a design in which the prioritization of high-value data is an integral part of the EMR, we decided to inform development with observations, interviews, and focus groups. We present paper prototypes for proposed design solutions that will inform the implementation of a functional user interface.

Methods

We combined the identification of relevant design themes in the literature with observations, interviews, and design activities with ICU physicians ([Fig. 1]).

Fig. 1 Study workflow. A non-systematic qualitative evidence synthesis provided insights on design principles applicable to a LEMR system, which inspired the creation of preliminary paper prototypes. To gain an understanding of information needs and practices in ICUs, we interviewed ICU physicians and observed their interactions with the current EMR system. A focus group of ICU physicians generated design ideas for the LEMR system and provided feedback on the concept and our prototypes, leading to the creation of a final series of prototypes, which will inform LEMR display development. ICU, intensive care unit; LEMR, Learning Electronic Medical Record.

Identification of Design Themes in the Literature and Preliminary Prototypes

Qualitative Evidence Synthesis and Prototype Development

We conducted a non-systematic qualitative evidence synthesis in January 2017, updating our search in July 2019 ([Fig. 2]). We limited our search to articles published between 1996 (year of publication of the first paper describing clinical data visualization techniques by Plaisant et al[7] [40]) and 2019, indexed in three databases (PubMed,[41] IEEE Xplore Digital Library,[42] ACM Digital Library[43]). We used a broad search string:

Fig. 2 Non-systematic qualitative evidence synthesis flow diagram. A non-systematic qualitative evidence synthesis provided insights on design principles applicable to a learning electronic medical record system designed for ICU care.

((EMR) OR (EHR)) AND ((data visualization) OR (dashboard) OR (design)) AND ((ICU) OR (intensive))

We utilized citation search and reference lists from overview papers written by information visualization experts[24] [44] [45] to find additional articles. Retrieved articles (n = 193) were screened by L.C. using title and abstract. To be selected, papers had to describe the use or development of ICU data visualization techniques, and/or provide design recommendations for the display of clinical data. We included literature reviews, and excluded papers describing interfaces not designed for clinical care, not focused on design aspects, not providing recommendations for the display of clinical data, or not published in English. H.H. independently repeated the selection process. Any disagreements were reconciled by consensus. Of 183 unique identified documents, 51 met the inclusion criteria (see [Supplementary Appendix A1] for full list [available in the online version]). L.C. and H.H. looked for insights on design aspects to be considered when displaying ICU data, grouped articles dealing with similar aspects, and used groupings to identify design themes applicable to a LEMR system. L.C. extracted data, coded all documents using emergent coding,[46] and developed a narrative synthesis. H.H. checked for accuracy of the extraction and validated the narrative. Differences were reconciled by consensus. Our 2017 search informed the creation of low-fidelity paper prototypes of the LEMR user interface. We designed all prototypes using Evolus Pencil, an open-source prototyping software tool,[47] and patient data from the HIgh DENsity Intensive Care (HIDENIC) dataset, containing fully de-identified, and HIPAA-compliant EMR data on University of Pittsburgh Medical Center ICU patients.[2]

Observations, Interviews, and Revised Prototypes

Participants

ICU fellows and attending physicians were contacted via email or personal contacts, and compensated using gift cards.

Observations

Author L.C. conducted observation sessions in multiple ICU settings in Pittsburgh, PA (Presbyterian, Children's, and Mercy Hospitals) from June to November 2017, observing ICU physicians using their EMRs (Cerner, in all cases) in preparation for morning rounds, and dealing with issues in finding information and workflow interruptions. Physician actions in patient rooms were not observed. Every observation was recorded using field notes.

Interviews

Observations were followed with semistructured interviews outside of the care setting. Participants were asked to explain interactions with the current EMR system, workflow and mental processes used to assess patients, and any interesting behaviors noticed. Additional questions addressed challenges encountered by the physicians while using the EMR system. All interviews were audio-recorded.

Data Analysis and Prototype Revision

Field notes and audio recordings were transcribed and independently coded by L.C. and H.H. using emergent coding.[46] L.C. coded all transcripts, developing a codebook with examples which HH used to recode one of the interview transcripts. Inter-rater agreement was calculated as a measure of coding consistency. Differences were reconciled by consensus to achieve high reliability (Cohen's Kappa > 0.98).[48] Codes were organized using QSR NVivo 12.[49] Extracted data were used to create a narrative model of cognitive processes underlying physicians' interactions with the EMR system, which informed the revision of our preliminary prototypes.

Focus Group and Final Prototypes

Participants

L.C. and H.H. conducted a focus group with fellows and attending physicians from multiple ICU settings in Pittsburgh, PA (Presbyterian, Children's, and Mercy Hospitals, VA Health System). Participants were contacted via individual emails or personal contacts, and compensated with gift cards.

Session

A 2-hour focus group session was conducted in the University of Pittsburgh Department of Critical Care Medicine in April 2018, using techniques inspired by participatory design[50]—a user-centered design approach that seeks to involve stakeholders in designing systems[51] [52] by combining brainstorming sessions with design activities.[53] After learning about our goal of designing a LEMR system and providing their feedback on the concept, participant groups of two to three members each brainstormed ideas about how the EMR display could highlight high-value information. Participants represented their ideas in low-fidelity sketches using craft materials. Physicians were asked to evaluate our prototypes and identify some preferred combinations of their ideas and ours. The session was audio-recorded, and artifacts and field notes were collected for later analysis.

Data Analysis and Final Prototypes

After the focus group, field notes, audio recordings, and discussions were transcribed, grouped by participant and group number, and independently coded by L.C. and H.H. using the techniques described in section 2.2.4. Any differences among raters were reconciled by consensus to achieve high reliability (Cohen's Kappa > 0.98).[48] Coded data, photographs, artifacts, and audio recordings were used to develop a narrative describing ICU physicians' feedback on the LEMR system and our prototypes, issues with the current EMR, and suggested design ideas. Data analysis informed the creation of a final series of paper prototypes, based on design themes and preferences on which all subjects unanimously agreed. Participants were asked to review and validate the narrative[54] and verify that the designs reflected their feedback.

Results

Design Themes Identified by Literature Review and Preliminary Prototypes

The design themes we identified in the literature as applicable to a LEMR system's interface are described below, together with examples of our resulting prototypes (the complete series is available in [Supplementary Appendix A2]; [Supplementary Figs. A2.1]–[A2.4] [available in the online version]).

Conveying Clinical Information Effectively

EMR interfaces should convey information effectively, by encoding health information using visual attributes,[24] prioritizing the display of high-value data,[10] [23] [30] [31] organizing information into clinical concepts,[24] [30] [32] and providing overviews of the patient's conditions.[2] [23] Our preliminary prototypes ([Fig. 3] and [Supplementary Appendix A2]; [Supplementary Figs. A2.1]–[A2.4] [available in the online version]) explored four approaches to the display of high-value information: highlighting information in place ([Supplementary Appendix A2]; [Supplementary Fig. A2.1] [available in the online version]), utilizing ephemeral highlighting to show initial in-place highlights that quickly fade,[55] using a reference map pointing to health information predicted to be of interest ([Supplementary Appendix A2]; [Supplementary Fig. A2.2] [available in the online version]), and utilizing a highlighted data panel ([Supplementary Appendix A2]; [Supplementary Fig. A2.3] [available in the online version]). To optimize screen information density,[10] [24] [27] [31] we also explored ways to summarize clinical information, including leveraging Midgaard's semantic zoom technique[27] to visualize variables at levels of detail that vary with the zoom level ([Fig. 3] and [Supplementary Appendix A2]; [Supplementary Fig. A2.4] [available in the online version]). Finally, in seeking to improve physician decision-making[30] and reduce cognitive load,[2] [56] we designed panels that provide overviews of the patient's conditions,[2] [23] organizing information into clinical concepts[24] [30] [32] and using perceptual attributes (color and shape) to facilitate data visualization[24] ([Supplementary Appendix A2]; [Supplementary Fig. A2.2] [available in the online version]).

Fig. 3 One of four preliminary paper prototypes: use of Midgaard's semantic zoom[27] to summarize clinical information, displaying a greater number of parameters at once. As displayed in the yellow box on the right (which was superimposed on the prototype for illustrative purposes and did not represent an actual component of the Learning Electronic Medical Record user interface), Midgaard's semantic zoom technique allows to visualize variables at levels of detail that vary with the zoom level.

Highlighting Trends and Changes in Clinical Outcomes

Patients showing unexpected trends and changes are prioritized by physicians.[22] We explored data visualizations that encode information on changes and trends,[24] such as arrows and triangles that indicate the presence of uptrends and downtrends[22] ([Supplementary Appendix A2]; [Supplementary Fig. A2.3] [available in the online version]).

Supporting Analytical Reasoning

EMR interfaces should support analytical reasoning[57] by visually grouping related data[36] and by facilitating the manipulation of information at the level of entities and their relationships.[21] [58] We considered grouping related laboratory test and medication data on the screen ([Supplementary Appendix A2]; [Supplementary Fig. A2.3] [available in the online version]). We also explored how linked selection, whereby selecting an entity in clinical notes could automatically highlight all instances of that entity in the EMR, might improve information retrieval[21] ([Supplementary Appendix A2]; [Supplementary Fig. A2.3] [available in the online version]).

Observations, Interviews, and Revised Prototypes

Five observation and interview sessions were conducted from June to November 2017 ([Table 1]). Insights from these interviews are presented below, together with one example of the resulting revised prototypes ([Fig. 4]). Representative direct quotes from participants are available in [Supplementary Appendix A3] [available in the online version]).

Table 1 Observations and focus group participant characteristics

Participant characteristics	Observations (n = 5)	Focus group (n = 5)
Gender
Male	2	3
Female	3	2
Age
Minimum	31	30
Average	32	31
Maximum	33	32
Years in clinical practice
Minimum	4	4
Average	5.4	5.6
Maximum	7	8
ICU team role
Fellows	5	5
Specialty
Surgery and critical care medicine	2	1
Cardiology and critical care medicine	1	–
Pediatrics and critical care medicine	1	–
Emergency and critical care medicine	1	1
Pulmonary and critical care medicine	–	1
Critical care medicine	–	2

Abbreviation: ICU, intensive care unit.

Fig. 4 One of four revised prototypes, showing how the Learning Electronic Medical Record interface might prioritize the display of (1) new information and (2) high-value patient data in dedicated panels that support analytical reasoning by (3) grouping related data, (4) highlighting changes and (5) trends, (6) providing unobtrusive alerts, and (7) augmenting clinical notes with links to related data items. For each parameter, the green color is used in the graphs to identify in-range values, while red and blue indicate values above or below the normal range, respectively.

Patient Assessment and Prioritization Process

Consistently with findings from previous studies,[1] [2] [3] physicians appeared to categorize patients using a preliminary mental schema and then to assess how well data fit with that schema. We observed that the presence of unexpected values, changes, and trends impacted patient prioritization and led physicians to gather more data.

Electronic Medical Record Data Usage

To assess patients in preparation for morning rounds, each physician visualized a limited subset of EMR data in a specific personalized order, influenced by observed trends and unexpected values. Some data were often visualized in the same sequence. In some circumstances, the need to document findings in the notes or to compare information while placing orders, forced users to repeatedly switch back and forth between EMR screens ([Supplementary Appendix A4]; [Supplementary Fig. A4.1] [available in the online version]).

Electronic Medical Record System Limitations

Participants expressed a need to reduce the amount of non-relevant information displayed on the EMR screens and a desire for a dashboard that concisely presents essential information, offering on-demand access to more details. The lack of a proper search feature makes it challenging to retrieve information in past notes. Visualizing frequently used data, such as ventilation settings or exams to be ordered, can require extensive scrolling or clicking. Integration with other information sources is inconsistent, and physicians are not alerted when specific pieces of information become available. The inability to tailor the system to individual needs also causes concern.

Workarounds

Workarounds, such as physician-created note templates and manual annotations on paper, are utilized to overcome limitations and data access difficulties. Rounding reports, index cards, and sign-out papers are used to annotate items that require discussion or attention, and to quickly look up information during chart review.

Feedback on the Learning Electronic Medical Record Concept

All participants expressed at the same time enthusiasm for the LEMR concept and prospected capabilities, and concern that users might overlook essential data if overrelying on system recommendations.

Focus Group and Final Prototypes

Five physicians from four ICU settings in Pittsburgh, PA participated in our focus group ([Table 1]). Two of them had previously participated in our observational study. Two senior physicians, including coauthor G.C., also attended part of the session. In the 2-hour session, physicians provided feedback on LEMR and our revised prototypes ([Supplementary Appendix A5]; [Supplementary Figs. A5.1]–[A5.5] [available in the online version]) and worked in two subgroups to generate design ideas, representing them using several artifacts ([Supplementary Appendix A6]; [Supplementary Figs. A6.1]–[A6.3] [available in the online version]). The subsequent discussion helped us reconcile overlapping and conflicting design themes emerged across the various stages of the study. Our final prototypes ([Fig. 4]; [Supplementary Appendix A7], [Supplementary Figs. A7.1] [A7.2] [available in the online version]) and the findings that guided their creation ([Table 2]) are presented below. Representative direct quotes from participants are available in [Supplementary Appendix A3] [available in the online version].

Table 2 The table summarizes: (1) the challenges posed by current electronic medical record systems that emerged from our qualitative evidence synthesis and were reconfirmed in the qualitative portion of the study, (2) the design principles identified in the literature as applicable to electronic medical record systems whose validity was reconfirmed by our study subjects, and (3) the novel design ideas emerged from our discussions with the participants (in bold, the proposed design augmentations that can be considered unique to LEMR-like systems)

Challenges posed by current EMR systems	Design principles applicable to EMR systems	Novel design ideas identified with study participants
		In bold, design ideas unique to LEMR-like systems
• Insufficient cognitive support provided to physicians • Limited capabilities for visualizing complex data that often complicate the identification of trends and anomalies • Fragmentation of information across multiple screens • Large amounts of non-relevant information displayed on screen • Lack of a proper search feature to retrieve information in past notes • Extensive scrolling or clicking required to visualize frequently used data • Inconsistent integration with other information sources • Unavailability of alerts when specific pieces of information become available • Inability to tailor the EMR system to individual needs	• Use of perceptual attributes (color and shape) to encode health information, including data on changes and trends. • Need to prioritize the display of high-value patient data, offering on-demand access to more details • Helpfulness of organizing information into clinical concepts • Need to optimize screen information density • Usefulness of panels that provide overviews of the patient's conditions, organizing information into clinical concepts • Need to support analytical reasoning by visually grouping related data	• Use of static screen components displaying data important for every patient, coupled with dynamic components highlighting diagnosis-specific information • Use of mouseovers or right-clicks on items to provide quick access to trend information and customizable views pertinent to the patient's active problems • Use of unobtrusive flags to highlight significant changes and trends in the data, and new data items (diagnostics, cultures, and select relevant labs) as they become available • Use of a “patient context” box to display information useful to characterize the patient's case • Use of a to-do list integrated with physician workflow, offering suggestions for frequent orders and the ability to manage orders/procedures and to track scheduled medication administrations • Use of design elements to provide physicians with the ability to give feedback on the appropriateness of the system recommendations

Abbreviations: ICU, intensive care unit; LEMR, Learning Electronic Medical Record.

Feedback on the Learning Electronic Medical Record Concept

The LEMR concept received unanimous support. Physicians expressed concern, however, that overreliance on the highlights might cause users to miss important information, suggesting that data not predicted to be of high value should remain easily accessible. Participants valued the ability to provide feedback on the appropriateness of the highlights.

Screen Layout and Access to Additional Information

Two alternative EMR home screen layouts were proposed. In [Fig. 5], high-value information pertinent to the patient's specific diagnoses could be displayed in a dedicated “highlighted data” box at the top of the screen (B). Alternatively ([Supplementary Appendix A7]; [Supplementary Fig. A7.1] [available in the online version]), static boxes displaying data important for every patient could be coupled with dynamic screen components highlighting diagnosis-specific information in place (e.g., lactate for a sepsis case). In both designs, boxes can be expanded when clicked upon to display additional data and visualized side by side to more easily compare information. Mouseovers or right-clicks on items provide quick access to trend information and customizable views pertinent to the patient's active problems ([Supplementary Appendix A7]; [Supplementary Fig. A7.3] [available in the online version]).

Fig. 5 One of two proposed electronic medical record home screen designs. Relevant data pertinent to the patient's specific diagnoses, (F) newly available and (D) significantly changed information, and (C, E) unobtrusive alerts are displayed in a dedicated “highlighted data” box at the top of the screen (B). Static boxes display (A, G, I, K, M–U) information important for every patient. Mouseovers or right-clicks on data items provide access to additional information. Blue indicators identify newly available data, while red indicators are used for alerts.

Display of Individual Electronic Medical Record Data Items

Reflecting participants' preferences, fishbones ([Fig. 5 N]) are used to display labs essential for every patient (e.g., basic metabolic panel, complete blood count, coagulation, and liver diagram). Mouseovers on labs provide access to trend information. In the graphs, expected normal ranges are shown by either bands or dotted lines showing upper and lower bounds. Time ranges default to 24 hours but can be expanded (H).

Highlighting Changes, Trends, and New Information

Color-coding out-of-range values was not seen as being helpful in an ICU. Physicians are more interested in significant changes and trends over time, which could be highlighted using unobtrusive flags ([Fig. 5 C, K]) to avoid information overload. Highlighting new data (diagnostics, cultures, and select relevant labs) as they become available, using color-coding (L) or a dedicated box (F) would also be helpful. Appropriately designed indicators for potential drug–drug interactions (C, K) and missed medication administrations (E) were also discussed as useful.

Display of Patient Context Information and Integration with Intensive Care Unit Workflow

Participants expressed the need for a “patient context” box ([Fig. 5A]) displaying information useful to characterize the patient's case: diagnoses, length of stay, unit the patient was transferred from, history and physical examination, presenting symptoms, assessment and plan, active problems, procedures, and links to relevant notes and consults. Using a to-do list ([Fig. 5P]) integrated with ICU workflow, physicians can visualize and cancel pending orders/procedures for the next 8 to 12 hours and add reminders. Scheduled medication administrations (with the ability to track if they were administered) and suggestions for frequent orders also appear on the to-do list.

Discussion

Advances Introduced by Our Work

The LEMR approach leverages EMR data access patterns (as instantiations of physicians' expertise) to predict and highlight the most relevant information for each patient. It is an approach to reducing information overload in the ICU that, to our knowledge, has not been considered before.[2] Our consideration of the effect of EMR interface design elements on cognitive performance[17] also contributes to the novelty of our approach.

Limitations of Current Electronic Medical Record Systems

Findings from observations and focus group were consistent with prior studies indicating that EMR systems do not offer adequate cognitive support to clinicians.[5] [6] [7] [8] Information overload and challenging access to information are major concerns[17]: multiple participants cited the overwhelming number of entries displayed in the medication ordering screens and the extensive scrolling/clicking required to access ventilation settings and other frequently used data. Data fragmentation also affects cognitive performance: by not allowing physicians to keep multiple windows simultaneously open, current EMR systems make it difficult to evaluate the patient's condition in its complexity.[17] To overcome these issues, physicians adopt workarounds such as custom note templates and manual annotations of items requiring discussion or attention. Our findings are consistent with prior studies, showing that workarounds are commonly associated with EMR use[17] [59] [60] [61] and introduce the risk of errors.[17] [62]

Identification and Display of High-Value Data

Our results suggest that information of high value to ICU physicians ([Fig. 6]) is represented by a combination of (1) data important for every patient with (2) diagnosis-related, patient-specific information, (3) significant changes and trends, and (4) newly available data (diagnostics, cultures, and select relevant labs).

Fig. 6 A combination of information essential for every patient, diagnosis-specific patient data, significant changes and trends, and select newly available information is predicted by the LEMR system to be of high-value, and displayed in a prioritized way in the LEMR interface by utilizing static and dynamic screen components. LEMR, Learning Electronic Medical Record.

Despite the complexity of the cognitive approaches ICU physicians use for sensemaking, the participants' artifacts expressed strikingly similar needs and solutions. Physicians identified approaches to highlighting high-value information that could enhance their cognitive performance: specifically, using dedicated panels and a combination of static and dynamic screen components that allow them to compare information and to access additional data via mouseovers ([Fig. 6]). Data presentations that encode changes and trends to visual attributes could enable easy identification of such information, while color-coding and obtrusive alerts should be minimized. To-do lists integrated with ICU workflow could offer cognitive support in the data-intensive ICU environment.[63]

Feedback on the Learning Electronic Medical Record System

All participants expressed enthusiasm for the LEMR system, confirming a need well-documented in the literature: EMR systems should provide better cognitive support to physicians.[21] [36] [39] Our design ideas were considered an acceptable approach to representing high-value data effectively.

Participants expressed concern that system use might introduce a form of automation bias, a “tendency to use automated cues as a heuristic replacement for active information seeking and processing”[64]. Physicians could become too reliant on the system's recommendations to identify information relevant to assess each patient, thus missing important data.[65] To address this concern, we are investigating how highlighting of predicted high-value items may impact physician information-search and interpretation activities. Participants suggested that non-highlighted data should always remain easily accessible.

Feedback on our designs suggested factors that physicians want to control when interacting with EMR systems. Users expressed a desire for customizability and for mechanisms to provide feedback on the appropriateness of the highlights. These desires could be reflective of the fear many physicians have of losing the “human element” of medicine[52] – in this case, having an electronic system make decisions for them.

Implementation Path and Challenges

We believe that the design preferences and prototypes identified in this study can usefully inform the implementation of a functional interface—initially as a standalone dashboard, to be used by clinicians as an addition to their current EMR systems. Increasingly, dashboards that provide access to high-value information in a visual, condensed format have been introduced by emergency departments[66] and health care organizations in general as ways of improving care processes and patient outcomes, with encouraging results.[67] We intend to build our dashboard using the Application Programming Interfaces relying on Fast Healthcare Interoperability Resources standards[68] proposed by major health care actors,[69] such as Epic[70] and Cerner.[71]

Developing a LEMR system display will involve several challenges. Designs must balance the potential benefits of highlighting high-value data items and supporting customization with the potential costs of related loss of consistency, which improves system learnability and facilitates locating information on screen.[72] Highlighting important and new information may seem appealing, but defining which changes are significant presents additional complications: further study is needed to compare general definitions based on the magnitude of a change (compared with overall parameter variance) to alternative approaches, such as definitions that take specific disease mechanisms into account. Careful consideration of information density principles will also be important:[10] while condensed views that summarize patient data represent a potential solution to the limited screen real estate available, denser views can easily lead to cognitive overload.[52]

Study Limitations

A potential limitation of our study is its single-center design. Despite the diversity of ICU settings within our sample, all participants worked at a single institution and used the same EMR system to accomplish a single task (chart review). Moreover, there is a chance we may not have captured the most representative feedback due to our exclusive focus on physician trainees and to the small number of subjects interviewed - even if, with only five participants in the observational study, the number of findings quickly reached the point of diminishing returns. There is also a chance that the preferences expressed by participants in a focus group setting might differ from those that they might express in clinical use.

Conclusion

Based on the positive feedback received from potential users, we conclude there is interest in pursuing the idea of a LEMR system. The findings of this study provide preliminary evidence of the potential utility of using highlights of clinical data predicted to be of high value as a potential means to deal with the problem of information overload associated with modern EMR systems. Further studies will be necessary to confirm the usefulness of our approach in a clinical setting. Future plans include the identification of ways to measure and present to physicians the confidence of the predictions generated by the system,[73] a usability evaluation with heuristics specifically designed for dashboard visualizations,[74] and laboratory studies of both the utility of the proposed designs on decision-making and the possible impact of any automation bias.

Clinical Relevance Statement

By introducing novel ways to support physicians' cognitive abilities in using EMR systems, our LEMR-based approach has the potential to enhance physician performance, leading to better patient outcomes as a result of those performance improvements.[17]

Multiple Choice Questions

1. What is the best argument in support of an EMR display that uses past information access patterns to predict and highlight particularly helpful information?

   • Information accessed when reviewing previous patients is a completely reliable model of information that will be needed on future patients.

   • High-quality models of past information access behavior provide accurate representations of access patterns over a large range of prior encounters, thus providing a useful representation of physicians' collective understanding of prior patients.

   • Information that was not accessed when reviewing prior similar patients is not likely to be useful.

   • Given two similar patients, all physicians will access exactly the same data when reviewing those patients.

   Correct Answer: The correct answer is option b. As information access patterns vary across physicians, no single model that does not highlight all information that was ever accessed by any physician can be completely reliable (thus ruling out answer a). As information access patterns tell us a great deal about what data physicians accessed, they do not tell us anything about information that physicians fail to access, so answer c cannot be correct. Finally, our small sample saw widely varying information access strategies, suggesting that identical access patterns are unlikely, making b the best answer.

2. When implementing an EMR system that predicts and highlights on screen the most relevant information for each patient, which of the following steps should be taken to avoid overreliance on the highlights?

   • Use of color-coding to highlight high-value information should be minimized.

   • Mechanisms to provide feedback on the appropriateness of the highlights should be made available to physicians.

   • Patient data not predicted to be of high value should remain easily accessible.

   • Access to additional patient data should be provided via mouseovers or right clicks on the highlights.

   Correct Answer: The correct answer is option c. The physicians participating in our study expressed concern that clinical use of an EMR system that predicts and highlights on screen the most relevant information for each patient might introduce a form of automation bias[64]: physicians could become too reliant on the system's recommendations, thus missing important data available elsewhere in the EMR.[65] To address this concern, participants suggested that data not predicted to be of high value should always remain easily accessible next to the highlights, making option c the correct answer.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the University of Pittsburgh Institutional Review Board (protocols 17030258, 17040082).

• References

• 1 Hall A, Walton G. Information overload within the health care system: a literature review. Health Info Libr J 2004; 21 (02) 102-108
• 2 King AJ, Cooper GF, Hochheiser H, Clermont G, Visweswaran S. Development and preliminary evaluation of a prototype of a Learning Electronic Medical Record system. AMIA Annu Symp Proc 2015; 2015: 1967-1975
• 3 King AJ, Hochheiser H, Visweswaran S, Clermont G, Cooper GF. Eye-tracking for clinical decision support: a method to capture automatically what physicians are viewing in the EMR. AMIA Jt Summits Transl Sci Proc 2017; 2017: 512-521
• 4 King AJ, Cooper GF, Hochheiser H, Clermont G, Hauskrecht M, Visweswaran S. Using machine learning to predict the information seeking behavior of clinicians using an electronic medical record system. AMIA Annu Symp Proc 2018; 2018: 673-682
• 5 Rind A. Interactive information visualization to explore and query Electronic Health Records. Found Trends Human–Computer Interact. 2013; 5 (03) 207-298
• 6 Stead WW, Lin HS. Computational Technology for Effective Health Care. Vol 22. Washington, D.C.: National Academies Press; 2009
• 7 West VL, Borland D, Hammond WE. Innovative information visualization of Electronic Health Record data: a systematic review. J Am Med Inform Assoc 2015; 22 (02) 330-339
• 8 Jaspers MWM, Peute LWP, Lauteslager A, Bakker PJM. Pre-post evaluation of physicians' satisfaction with a redesigned electronic medical record system. Stud Health Technol Inform 2008; 136: 303-308
• 9 Sittig DF, Murphy DR, Smith MW, Russo E, Wright A, Singh H. Graphical display of diagnostic test results in electronic health records: a comparison of 8 systems. J Am Med Inform Assoc 2015; 22 (04) 900-904
• 10 Armijo D, McDonnell C, Werner K. Electronic Health Record Usability: Interface Design Considerations. Vol 09. Rockville, MD; 2009
• 11 Bawden D. Perspectives on information overload. Aslib Proc 1999; 51 (08) 249-255
• 12 Gross B. The Managing of Organizations: The Administrative Struggle. New York: Free Press of Glencoe; 1964
• 13 Food and Drug Administration (FDA). FDASIA health IT report. Proposed strategy and recommendations for a risk-based framework 2014. Accessed March 26, 2020 at: https://www.fda.gov/about-fda/cdrh-reports/fdasia-health-it-report
• 14 Thongprayoon C, Harrison AM, O'Horo JC, Berrios RAS, Pickering BW, Herasevich V. The effect of an electronic checklist on critical care provider workload, errors, and performance. J Intensive Care Med 2016; 31 (03) 205-212
• 15 Patel VL, Currie LM. Clinical cognition and biomedical informatics: issues of patient safety. Int J Med Inform 2005; 74 (11-12): 869-885
• 16 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14 (04) 415-423
• 17 Holden RJ. Cognitive performance-altering effects of electronic medical records: An application of the human factors paradigm for patient safety. Cogn Technol Work 2011; 13 (01) 11-29
• 18 Pickering BW, Gajic O, Ahmed A, Herasevich V, Keegan MT. Data utilization for medical decision making at the time of patient admission to ICU. Crit Care Med 2013; 41 (06) 1502-1510
• 19 Wright MC, Dunbar S, Macpherson BC. et al. Toward designing information display to support critical care. a qualitative contextual evaluation and visioning effort. Appl Clin Inform 2016; 7 (04) 912-929
• 20 Kannampallil TG, Jones LK, Patel VL, Buchman TG, Franklin A. Comparing the information seeking strategies of residents, nurse practitioners, and physician assistants in critical care settings. J Am Med Inform Assoc 2014; 21 (e2): e249-e256
• 21 Kannampallil TG, Franklin A, Mishra R, Almoosa KF, Cohen T, Patel VL. Understanding the nature of information seeking behavior in critical care: implications for the design of health information technology. Artif Intell Med 2013; 57 (01) 21-29
• 22 Pollack AH, Tweedy CG, Blondon K, Pratt W. Knowledge crystallization and clinical priorities: evaluating how physicians collect and synthesize patient-related data. AMIA Annu Symp Proc 2014; 2014: 1874-1883
• 23 Pickering BW, Herasevich V, Ahmed A, Gajic O. Novel representation of clinical information in the ICU: developing user interfaces which reduce information overload. Appl Clin Inform 2010; 1 (02) 116-131
• 24 Ware C. Guidelines. In: Information Visualization. Cambridge, MA: Elsevier; 2013: 445-457
• 25 Faiola A, Newlon C. Advancing critical care in the ICU: a human-centered biomedical data visualization system. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol 6779 LNCS. Indianapolis, IN: Indiana University, School of Informatics; 2011: 119-128
• 26 Horn W, Popow C, Unterasinger L. Support for fast comprehension of ICU data: visualization using metaphor graphics. Methods Inf Med 2001; 40 (05) 421-424
• 27 Bade R, Schlechtweg S, Miksch S. Connecting time-oriented data and information to a coherent interactive visualization. In: Proceedings of the 2004 Conference on Human Factors in Computing Systems - CHI '04. Vol 6. New York, New York, USA: ACM Press; 2004: 105-112 . Accessed March 26, 2020 at: https://dl.acm.org/doi/10.1145/985692.985706
• 28 Klimov D, Shahar Y, Taieb-Maimon M. Intelligent visualization and exploration of time-oriented data of multiple patients. Artif Intell Med 2010; 49 (01) 11-31
• 29 Spenke M. Visualization and interactive analysis of blood parameters with InfoZoom. Artif Intell Med 2001; 22 (02) 159-172
• 30 Pickering BW, Dong Y, Ahmed A. et al. The implementation of clinician designed, human-centered electronic medical record viewer in the intensive care unit: a pilot step-wedge cluster randomized trial. Int J Med Inform 2015; 84 (05) 299-307
• 31 Wang TD, Wongsuphasawat K, Plaisant C, Shneiderman B. Visual information seeking in multiple electronic health records. In: Proceedings of the ACM International Conference on Health Informatics - IHI '10. New York, New York, USA: ACM Press; 2010: 46 . Accessed March 26, 2020 at: https://dl.acm.org/doi/10.1145/1882992.1883001
• 32 Zeng Q, Cimino JJ. Evaluation of a system to identify relevant patient information and its impact on clinical information retrieval. Proc AMIA Symp 1999; 642-646
• 33 Pieczkiewicz DS, Finkelstein SM, Hertz MI. Design and evaluation of a web-based interactive visualization system for lung transplant home monitoring data. AMIA Annu Symp Proc 2007; 598-602
• 34 Mamykina L, Vawdrey DK, Stetson PD, Zheng K, Hripcsak G. Clinical documentation: composition or synthesis?. J Am Med Inform Assoc 2012; 19 (06) 1025-1031
• 35 Shahar Y, Goren-Bar D, Boaz D, Tahan G. Distributed, intelligent, interactive visualization and exploration of time-oriented clinical data and their abstractions. Artif Intell Med 2006; 38 (02) 115-135
• 36 Senathirajah Y, Kaufman D, Bakken S. Beyond copy and paste: clinician approaches to meeting information needs during note writing. Stud Health Technol Inform 2014; 205: 599-603
• 37 van Merriënboer JJG, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ 2010; 44 (01) 85-93
• 38 Ahmed A, Chandra S, Herasevich V, Gajic O, Pickering BW. The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance. Crit Care Med 2011; 39 (07) 1626-1634
• 39 Karsh B-T, Holden RJ, Alper SJ, Or CKLL. A human factors engineering paradigm for patient safety: designing to support the performance of the healthcare professional. Qual Saf Health Care 2006; 15 (Suppl. 01) i59-i65
• 40 Plaisant C, Milash B, Rose A, Widoff S, Shneiderman B. LifeLines: visualizing personal histories. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Common Ground - CHI '96. New York, New York, USA: ACM Press; 1996 :221-ff
• 41 National Center for Biotechnology Information (US). PubMed Help. NCBI Bookshelf. Published 2014. Accessed December 12, 2018 at: https://www.ncbi.nlm.nih.gov/books/NBK3827/
• 42 Institute of Electrical and Electronics Engineers. About IEEE Xplore® Digital Library. Accessed July 19, 2019 at: https://ieeexplore.ieee.org/xpl/aboutUs.jsp
• 43 Association for Computing Machinery. ACM digital library. Published 2019. Accessed July 19, 2019 at: https://dl.acm.org/
• 44 Tufte ER. The visual display of quantitative information. IEEE Power Eng Rev 1988; 8 (02) 20-20
• 45 Aigner W, Miksch S, Schumann H, Tominski C. Visualization of time-oriented data. 2011
• 46 Kzillen M, Jarrett N. Qualitative data analysis: an introduction. J Adv Nurs 2007; 59 (05) 557-557
• 47 Evolution Solutions Co. Ltd. Pencil Project. Published 2016. Accessed April 28, 2016 at: http://pencil.evolus.vn/
• 48 McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012; 22 (03) 276-282
• 49 QSR. NVivo 11: Research software for analysis and insight. Published 2016. Accessed December 31, 2016 at: http://www.qsrinternational.com/
• 50 Muller MJ. PICTIVE - an exploration in participatory design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Reaching through Technology - CHI '91. New York, New York, USA: ACM Press; 1991: 223-225
• 51 Estiri H, Lovins T, Afzalan N, Stephens KA. Applying a participatory design approach to define objectives and properties of a “data profiling” tool for electronic health data. AMIA Jt Summits Transl Sci Proc 2016; 2016: 60-67
• 52 Rose AF, Schnipper JL, Park ER, Poon EG, Li Q, Middleton B. Using qualitative studies to improve the usability of an EMR. J Biomed Inform 2005; 38 (01) 51-60
• 53 Pollack AH, Miller A, Mishra SR, Pratt W. PD-atricians: leveraging physicians and Participatory Design to develop novel clinical information tools. AMIA. Annu Symp proceedings AMIA Symp. 2016;2016:1030–1039 Accessed March 26, 2020 at: https://www.ncbi.nlm.nih.gov/pubmed/28269900
• 54 Lincoln YS, Guba EG. Naturalistic Inquiry. Vol 5. Beverly Hills, CA: Sage Publications; 1985
• 55 Findlater L, Moffatt K, McGrenere J, Dawson J. Ephemeral adaptation. In: Proceedings of the 27th International Conference on Human Factors in Computing Systems - CHI 09. New York, New York, USA: ACM Press; 2009: 1655
• 56 Asan O, Holden RJ, Flynn KE, Yang Y, Azam L, Scanlon MC. Provider use of a novel EHR display in the pediatric intensive care unit. large customizable interactive monitor (LCIM). Appl Clin Inform 2016; 7 (03) 682-692
• 57 Zheng K, Hanauer DA, Weibel N, Agha Z, Cognitive Informatics for Biomedicine. In: Patel VL, Kannampallil TG, Kaufman DR. , eds. Cham: Springer International Publishing; 2015
• 58 Youn-Ah Kang, Görg C, Stasko J. How can visual analytics assist investigative analysis? Design implications from an evaluation. IEEE Trans Vis Comput Graph 2011; 17 (05) 570-583
• 59 Novak LL, Holden RJ, Anders SH, Hong JY, Karsh B-T. Using a sociotechnical framework to understand adaptations in health IT implementation. Int J Med Inform 2013; 82 (12) e331-e344
• 60 Patterson ES, Cook RI, Render ML. Improving patient safety by identifying side effects from introducing bar coding in medication administration. J Am Med Inform Assoc 2002; 9 (05) 540-553
• 61 Koppel R, Wetterneck T, Telles JL, Karsh B-T. Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety. J Am Med Inform Assoc 2008; 15 (04) 408-423
• 62 Harrison MI, Koppel R. Interactive sociotechnical analysis. In: Handbook of Research on Advances in Health Informatics and Electronic Healthcare Applications. PA, USA: IGI Global; 1AD:33-51
• 63 Rivera-Rodriguez AJ, Karsh B-T. Interruptions and distractions in healthcare: review and reappraisal. Qual Saf Health Care 2010; 19 (04) 304-312
• 64 Mosier KL, Skitka LJ, Parasuraman R, Mouloua M. Human decision makers and automated decision aids: made for each other?. In: Automation and Human Performance: Theory and Applications. Erlbaum; 1996: 201-220
• 65 Ancker JS, Kern LM, Edwards A. , et al; HITEC Investigators. How is the electronic health record being used? Use of EHR data to assess physician-level variability in technology use. J Am Med Inform Assoc 2014; 21 (06) 1001-1008
• 66 Rasmussen R. Electronic whiteboards in emergency medicine: a systematic review. In: IHI'12 - Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium.; 2012: 483-492
• 67 Dowding D, Randell R, Gardner P. et al. Dashboards for improving patient care: review of the literature. Int J Med Inform 2015; 84 (02) 87-100
• 68 HL7 FHIR Foundation. HL7 FHIR. Accessed January 4, 2020 at: https://www.hl7.org/fhir/
• 69 Giordanengo A, Årsand E, Woldaregay AZ. et al. Design and prestudy assessment of a dashboard for presenting self-collected health data of patients with diabetes to clinicians: Iterative approach and qualitative case study. JMIR Diabetes 2019; 4 (03) e14002
• 70 Epic Systems Corporation. Epic App Orchard. Accessed January 4, 2020 at: https://apporchard.epic.com/
• 71 Cerner Corporation. Leverage the power of the HL7® FHIR® standard in your SMART app. Accessed January 4, 2020 at: https://fhir.cerner.com/
• 72 Nielsen J. Usability Engineering. San Francisco, CA: Morgan Kaufmann; 1994
• 73 Tajgardoon M, Samayamuthu MJ, Calzoni L, Visweswaran S. Patient-Specific Explanations for Predictions of Clinical Outcomes. ACI Open 2019; 03 (02) e88-e97
• 74 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518

Publication History

Received: 17 October 2019

Accepted: 09 March 2020

Article published online:
14 October 2020