Subscribe to RSS
DOI: 10.1055/s-0040-1701255
Promoting Quality Face-to-Face Communication during Ophthalmology Encounters in the Electronic Health Record Era
Address for correspondence
Publication History
06 November 2019
24 December 2019
Publication Date:
19 February 2020 (online)
- Abstract
- Background and Significance
- Objectives
- Methods
- Results
- Discussion
- Conclusion
- Clinical Relevance Statement
- Multiple Choice Questions
- References
Abstract
Objective To evaluate informatics-enabled quality improvement (QI) strategies for promoting time spent on face-to-face communication between ophthalmologists and patients.
Methods This prospective study involved deploying QI strategies during implementation of an enterprise-wide vendor electronic health record (EHR) in an outpatient academic ophthalmology department. Strategies included developing single sign-on capabilities, activating mobile- and tablet-based applications, EHR personalization training, creating novel workflows for team-based orders, and promoting problem-based charting to reduce documentation burden. Timing data were collected during 648 outpatient encounters. Outcomes included total time spent by the attending ophthalmologist on the patient, time spent on documentation, time spent on examination, and time spent talking with the patient. Metrics related to documentation efficiency, use of personalization features, use of team-based orders, and note length were also measured from the EHR efficiency portal and compared with averages for ophthalmologists nationwide using the same EHR.
Results Time spent on exclusive face-to-face communication with patients initially decreased with EHR implementation (2.9 to 2.3 minutes, p = 0.005) but returned to the paper baseline by 6 months (2.8 minutes, p = 0.99). Observed participants outperformed national averages of ophthalmologists using the same vendor system on documentation time per appointment, number of customized note templates, number of customized order lists, utilization of team-based orders, note length, and time spent after-hours on EHR use.
Conclusion Informatics-enabled QI interventions can promote patient-centeredness and face-to-face communication in high-volume outpatient ophthalmology encounters. By employing an array of interventions, time spent exclusively talking with the patient returned to levels equivalent to paper charts by 6 months after EHR implementation. This was achieved without requiring EHR redesign, use of scribes, or excessive after-hours work. Documentation efficiency can be achieved using interventions promoting personalization and team-based workflows. Given their efficacy in preserving face-to-face physician–patient interactions, these strategies may help alleviate risk of physician burnout.
#
Keywords
electronic health records - ophthalmology - quality improvement - informatics - communicationBackground and Significance
Burnout among physicians is increasing,[1] [2] affecting nearly 80% of American physicians according to a national survey in 2018.[3] While the causes of physician burnout are complex and multifactorial, increasing use of electronic health records (EHRs) has been frequently cited as a contributing factor.[1] [4] [5] [6] [7] [8] [9] [10] Although documentation in EHRs has been demonstrated to be more complete than paper-based documentation,[11] EHRs have also been associated with increased clerical burden and decreased face-to-face time with patients due to computer use.[1] [4] [7] [12] [13] [14] [15] [16] One longitudinal study conducted in an academic ophthalmology department over 2 years before and after EHR implementation demonstrated that ophthalmologists were significantly more likely to express concern about the impact of an EHR on meaningful patient interaction after implementation.[17] One strategy to address these issues has focused on redesigning EHRs, with multiple studies examining how to improve EHR usability, optimize user interfaces, and reduce click burden.[8] [18] [19] [20] [21] [22] [23] However, executing EHR design changes can be constrained by limitations imposed by institutional infrastructure or by EHR vendors. Another strategy to reduce the documentation burden and altered physician–patient communication patterns imposed by EHRs is to use scribes. Scribes have been shown to improve physician satisfaction, charting efficiency, productivity, and face-to-face communication with patients.[24] [25] [26] [27] [28] However, some have suggested that using scribes results in lost nuance and cognitive processing time,[29] and relatively little is known about the safety and accuracy of scribe-related documentation.[30] Moreover, scribes may not always be financially feasible. Therefore, additional strategies to improve EHR use warrant further investigation.
#
Objectives
In this study, we evaluated whether a suite of informatics-enabled quality improvement (QI) interventions could optimize documentation efficiency and face-to-face communication during outpatient encounters. Ophthalmology serves as a representative use case as it entails both medical and surgical practice, involves clinical workflows similar to many other ambulatory specialties (initial staff exam followed by clinician), and tends to be high volume.[31] We leveraged a paper-to-EHR transition in an academic ophthalmology department as an opportunity to study the efficacy of QI interventions deployed during the implementation process. The efficacy of these interventions was evaluated by comparing timing outcomes for EHR use with those for paper charts, which have been previously reported.[32] Based on prior studies,[33] [34] [35] we anticipated that EHR use would be associated with significantly increased documentation time and decreased face-to-face communication time.
We hypothesized that these anticipated effects could be mitigated by a multipronged approach: (1) developing single sign-on (SSO) capabilities; (2) activating mobile- and tablet-based applications; (3) training on existing EHR personalization features; (4) creating novel workflows for team-based orders; and (5) focusing on mitigating documentation burden. Here we demonstrate that this approach enabled the time spent on face-to-face communication during outpatient ophthalmology encounters using EHRs to reach equivalence to paper charting, even without scribes. Although these interventions were studied in the context of an EHR implementation, they leveraged existing EHR functionalities and could also be operationalized immediately for clinical settings that are already using EHR. With increasing EHR adoption among ophthalmologists,[36] these types of interventions will become increasingly relevant and offer potential strategies for reducing EHR dissatisfaction and preventing associated burnout.
#
Methods
Study Design and Subjects
This was a prospective study at the University of California San Diego (UCSD) Shiley Eye Institute, a multispecialty academic ophthalmology clinic in La Jolla, California, United States. Prior to September 2018, all outpatient encounters were documented using paper charts. A convenience sample of outpatient encounters in clinics of attending ophthalmologists who underwent implementation of the ophthalmology module of the enterprise EHR (Kaleidoscope from Epic Systems, Verona, Wisconsin, United States) in September 2018 was used to obtain manual time–motion observations before and after implementation. Institutional review board/ethics committee approval was obtained. Waiver of documented consent was granted to allow observation of patients and ophthalmologists without disruption of normal workflows.
#
Intervention Strategies
Building on the national experience and published literature around EHR optimization,[37] [38] [39] the following strategies were employed to decrease ophthalmologists' documentation burden and improve their ability to engage in patient-facing activities.
Developing Single Sign-On Capabilities
To reduce click burden, secure SSO capability was installed for all workstations (Imprivata, Inc., Lexington, Massachusetts, United States). This allowed simultaneous access to the workstation desktop, the enterprise EHR, and the ophthalmology picture and archiving communication system (PACS; ZEISS Forum Enterprise, Carl Zeiss Meditec, Oberkochen, Germany) with a single-badge contact to a proximity card reader. Not only was this process keyboard and mouse-free, but it also allowed for simultaneous navigation between the EHR and the PACS as ophthalmologists entered different patient charts. In addition, functionality was developed so that ophthalmologists badging into a workstation would automatically open the last patient chart that was active on that workstation, which was typically prepopulated by a technician and/or a trainee. This eliminated the need to scroll through the clinic schedule to select a particular patient's chart at every log-in.
#
Activating Mobile- and Tablet-Based Applications
To allow greater flexibility of EHR use and improve the ophthalmologists' ability to turn and face the patient instead of being obligated to face a static computer screen, training was also provided on how to use the EHR on mobile- and tablet-based applications (Epic Haiku and Epic Canto, Epic Systems).
#
Training on Existing EHR Personalization Features
The enterprise EHR allowed users to customize a wide range of features, including note templates, order sets, and menus/displays. Multiple methods were employed to train ophthalmologists, technicians, and trainees about these features. EHR analysts with expertise of the ophthalmology module conducted formal 4-hour training sessions for all users, divided by user group and by subspecialty division to customize each session to specific workflow needs. These sessions were not wholly didactic; each had 1 to 2 hour built-in time for users to develop personalization features with analysts' assistance. Additionally, individual consultations from EHR analysts, the ambulatory Chief Medical Information Officer (M.M.), and a dedicated liaison between the ophthalmology department and the health information technology (IT) team (H.E.G.) were also offered to all ophthalmologists for further guidance.
#
Creating Novel Workflows for Team-Based Orders
Recognizing that EHR adoption could shift clerical responsibilities to physicians, the implementation team developed and implemented novel workflows for team-based orders within the EHR, leveraging ophthalmic technicians who had been accustomed to offering substantial support to ophthalmologists in paper-based workflows. The ophthalmology module allowed technicians to pend orders for imaging and other ancillary tests. In addition, for tests that were likely to be repeated over time (i.e., visual fields), technicians could easily pend repeating orders—once signed, these orders would automatically activate for future encounters. Technicians provided input as the workflow was being developed, and multiple “dress rehearsals” were conducted prior to EHR implementation to refine the workflows.
#
Documentation Burden
Lengthy physician notes as a result of billing, coding, and regulatory requirements, as well as conservative local interpretation of these guidelines, may contribute substantially to EHR-related work burden.[40] [41] In close collaboration with local compliance experts, the team built innovative problem-based documentation workflows to reduce data entry burden and shorten note length. Providers were able to document and order all in one screen using problem-based documentation. They documented results of imaging in their notes and avoided needing to navigate to other areas of the chart. In addition, a new type of note was created to make the visit information easily visible to other providers as well as formatted to be billed.
#
#
Main Outcome Measures: Timing Metrics
Timing Metrics Based on Manual Time–Motion Observations
The main outcome measures consisted of timing metrics for each patient encounter. Detailed timestamp data were collected using manual time–motion observations regarding total time spent by the ophthalmologist on the encounter, time spent on documentation (which was broadly defined to include any activities entailing interaction with the patient's medical chart), time spent on physical examination, and time spent talking with the patient. Time spent on documentation included time spent interacting with the patient's medical chart whether on paper, desktop-based EHR, EHR mobile application, or EHR tablet application. All other activities (i.e., performing procedures, talking with trainees or staff about the patient) were categorized as “other.” Consistent with previously published methodology,[32] [33] if talking with the patient occurred simultaneously with another activity, only the nontalking activity was recorded. Therefore, “talking with the patient” was only recorded if the ophthalmologist was engaged in exclusive, face-to-face communication with the patient. Timing data were recorded using customized data entry tools (Numbers, Apple Inc., Cupertino, California, United States) with automated timestamp functionality and preloaded menus with specified activities to facilitate rapid data entry and decrease interobserver variability. Training and quality control measures for manual time–motion observations in the outpatient ophthalmology setting have been previously described.[32] In summary, paired observations were conducted to ensure interobserver consistency in accurate timing and identification of clinical activities.
Manual time–motion observations were performed for 648 patient encounters. All patient encounters in clinics of full-time faculty members with stable practices (defined as working >1 year at UCSD with four or more half-day clinic sessions per month) who were undergoing EHR implementation were eligible for inclusion. Time–motion observations were performed in three phases: (1) 2 to 3 weeks before EHR implementation in September 2018, with paper-based documentation (“preimplementation”); (2) 5 to 6 weeks after EHR implementation in November 2018 (“early postimplementation”); and (3) 6 months after EHR implementation in April 2019 (“6 months postimplementation”). In a systematic review examining the impact of EHR use on time efficiency,[42] the most common time period from implementation to evaluation was 6 to 7 months. Prior time–motion studies in ophthalmology also evaluated EHR use at 5 to 6 months after implementation.[34] [35] We therefore selected 6 months as an evaluation time point to enable comparison with prior studies. We also evaluated timing outcomes at 6 weeks postimplementation to characterize early EHR use. Manual time–motion observations required a physically present observer and therefore precluded masking or blinding procedures. For each patient, demographic data (age, gender, and ethnicity), primary language used during the encounter, and visit type (new patient, routine follow-up, and postoperative visit within 90 days of surgery) were recorded. This allowed comparison of patient characteristics across study phases.
#
Timing Metrics Based on the EHR Efficiency Portal
We also analyzed data from an efficiency portal within the EHR (Epic Signal, Epic Systems) to characterize usage metrics among observed ophthalmologists regarding efficiency and adoption of EHR features after these intervention strategies had been implemented. We used readily available usage metrics that were developed by the EHR vendor and accessible on the efficiency portal, which is a web-based interface that reports the usage metrics alongside an array of visualization tools. These metrics, aggregated by monthly reporting periods, are based on vendor-designed algorithms that process timestamped EHR data and map them into specific domains such as notes, orders, clinical review, in-basket time, and other activities. We collected EHR timing metrics for ophthalmologists in our study in the April 2019 reporting period, which was the same time period as the final phase of time–motion observations (6 months post-EHR implementation). These were compared with aggregated, deidentified data reported for all ophthalmologists using the same EHR vendor nationwide during the same reporting period.
#
#
Process Measures
Because patient volume has been shown to impact EHR use time, the first process measure was clinical volume, defined as the number of patient encounters completed by each observed ophthalmologist within a half-day clinic session, defined by their usual scheduling template. For each session where manual time–motion observations were performed, the total number of completed patient encounters was also recorded. Patients that had been scheduled but did not arrive for their appointments (no-shows) were excluded. The mean clinical volume was calculated across all observed sessions for all ophthalmologists in each study phase.
Besides clinical volume, other process measures described the adoption of the intervention strategies described. Direct observation was used to determine whether ophthalmologists used the SSO/badge-in functionality or manually typed in their log-in credentials, and whether they used mobile and tablet EHR applications. The EHR efficiency portal was used to collect data on whether observed ophthalmologists adopted various personalization features in the EHR, what percentage of orders had team contributions, and how long their notes were based on character count. These data helped generate insight about which processes may have contributed to the observed effects on timing outcomes.
#
Balancing Measures
Balancing measures reflect perceptions of a system from different dimensions and are aimed at quantifying possible new problems in the system.[43] As such, balancing measures help elucidate whether changes designed to improve one part of a system may cause new problems in other parts of the system. In this study, our balancing measure consisted of patient satisfaction scores. Patient satisfaction scores were obtained using routinely administered Press Ganey surveys that ask patients whether they would “recommend the physician and service.”[44] We defined patient satisfaction as the percentage of patients who responded “Yes, definitely.” Patient satisfaction scores were obtained in August 2018, as this represented the final full month of paper-based documentation, before EHR implementation in September 2019. This was averaged across all observed ophthalmologists and considered the baseline patient satisfaction value. Subsequent monthly averages across all observed ophthalmologists throughout the duration of the study period were reported relative to the baseline value (i.e., each average monthly value was divided by the baseline value).
#
Statistical Analysis
The differences between continuous timing outcomes from the study phases were analyzed using the Kruskal–Wallis rank sum tests. If the Kruskal–Wallis rank sum test demonstrated a statistically significant difference between study phases, subsequent multiple pairwise comparisons were performed between groups using pairwise Wilcoxon rank sum tests. We used the false discovery rate method to correct for multiple comparisons. Differences in clinical volume across study phases were evaluated using analysis of variance (ANOVA) testing. Statistical significance was defined as p < 0.05 for all hypothesis tests. Besides timing outcomes and clinical volume, descriptive statistics were generated for other measures, but hypothesis testing was not performed given that aggregated data in the EHR efficiency portal did not provide measures of variance. All statistical analyses were performed using R (The R Project for Statistical Computing).[45]
#
#
Results
Cohort Characteristics
Time–motion observations were performed for 648 encounters (n = 227 preimplementation, n = 170 at 5–6 weeks after EHR implementation, and n = 251 at 6 months after EHR implementation). The baseline characteristics of observed patients were similar across the three study phases ([Table 1]). Among the four ophthalmologists whose clinics were observed across all the three study phases, the mean (standard deviation, SD) age was 43.5 (8.1) years. The distribution of ophthalmologists by subspecialty division was one (25%) comprehensive, one (25%) glaucoma, and two (50%) oculoplastics. Two (50%) were full professors, one (25%) was an associate professor, and one (25%) was an assistant professor. All had prior experience using the enterprise EHR for inpatient consultations and in the ambulatory surgical suite, which had implemented the EHR 4 years prior.
Preimplementation (n = 227) |
Early postimplementation[a] (n = 170) |
6 months postimplementation (n = 251) |
|
---|---|---|---|
Age |
|||
Mean (SD) |
63.2 (18.8) |
63.5 (17.6) |
59.2 (20.0) |
Gender |
|||
Female |
136 (59.9%) |
97 (57.1%) |
153 (61.0%) |
Male |
91 (40.1%) |
70 (41.2%) |
98 (39.0%) |
Ethnicity |
|||
White or Caucasian |
139 (61.2%) |
97 (57.1%) |
146 (58.2%) |
Black or African American |
8 (3.5%) |
7 (4.1%) |
10 (4.0%) |
Asian |
40 (17.6%) |
27 (15.9%) |
42 (16.7%) |
Latino or Hispanic |
30 (13.2%) |
20 (11.8%) |
44 (17.5%) |
Other |
10 (4.4%) |
16 (9.4%) |
9 (3.6%) |
Language |
|||
English |
206 (90.7%) |
150 (88.2%) |
221 (88.0%) |
Spanish |
9 (4.0%) |
7 (4.1%) |
20 (8.0%) |
Other |
12 (5.3%) |
10 (5.9%) |
10 (4.0%) |
Visit type |
|||
New patient evaluation |
62 (27.3%) |
42 (24.7%) |
61 (24.3%) |
Routine follow-up |
96 (42.3%) |
96 (56.5%) |
143 (57.0%) |
Postoperative (within 90 days) |
69 (30.4%) |
29 (17.1%) |
47 (18.7%) |
Subspecialty division |
|||
Comprehensive |
62 (27.3%) |
36 (21.2%) |
69 (27.5%) |
Glaucoma |
50 (22.0%) |
55 (32.4%) |
43 (17.1%) |
Oculoplastics |
115 (50.7%) |
79 (46.5%) |
139 (55.4%) |
Note: Different patients were examined across the phases.
a Numbers and percentages do not add up to 100% due to three patients with missing demographic data in the early postimplementation phase.
#
Outcome Measures: Timing Metrics
Timing Outcomes Based on Time–Motion Observations
Distributions of timing outcomes based on manual time–motion observations are illustrated in [Fig. 1], while [Table 2] details key timing outcomes across all study phases based on manual time–motion observations. The time spent exclusively talking with patients initially decreased from 2.9 (2.4) minutes on paper charts to 2.3 (2.2) minutes early postimplementation (p = 0.005). This corresponded to a decrease in the proportion of the total encounter devoted to talking with the patient from 40.8 to 25.8% ([Fig. 2]). However, time spent talking with the patient subsequently increased to 2.8 (2.3) minutes (34.1%) at 6 months postimplementation (p = 0.005, when compared with early postimplementation; [Fig. 2]). After 6 months of EHR use, the time spent on exclusive face-to-face communication with patients was not significantly different when compared with paper charts (p = 0.99).
Preimplementation (n = 227) |
Early postimplementation (n = 170) |
6 months postimplementation (n = 251) |
|
---|---|---|---|
Total time per patient |
7.1 (3.9) |
8.9 (4.7) |
8.2 (4.6) |
Documentation time per patient |
2.1 (1.3) |
4.1 (2.1) |
3.3 (2.0) |
Examination time per patient |
1.2 (0.8) |
1.0 (0.8) |
1.2 (1.1) |
Talking time per patient |
2.9 (2.4) |
2.3 (2.2) |
2.8 (2.3) |
a Values are reported as mean (standard deviation) time in minutes.
The mean (SD) documentation time increased from 2.1 (1.3) minutes on paper to 4.1 (2.1) minutes early postimplementation (p < 0.001). However, by 6 months postimplementation, the mean (SD) documentation time decreased to 3.3 (2.0) minutes, which was significantly less than the documentation time early postimplementation (p < 0.001) but still significantly more than the paper baseline (p < 0.001). Similarly, the proportion of the total encounter devoted to documentation increased from 29.6% on paper charts to 46.1% in the early postimplementation phase, then decreased to 40.2% by 6 months after EHR implementation ([Fig. 2]). Time spent examining patients was stable across all phases of the study (p = 0.09).
#
Timing Outcomes Based on the EHR Efficiency Portal
Based on EHR usage metrics provided by the vendor's efficiency platform, observed ophthalmologists spent less time in orders, notes, and clinical review per appointment compared with the average values of ophthalmologists nationwide using the same EHR at 6 months postimplementation ([Table 3]). Moreover, ophthalmologists in this study who had undergone the aforementioned intervention strategies spent less time outside scheduled hours, less time on unscheduled days, and less time outside 7 a.m. to 7 p.m. compared with national averages ([Table 3]).
Note: Average values for observed ophthalmologists undergoing targeted intervention strategies during EHR implementation in September 2018 compared with ophthalmologists nationwide using the same EHR vendor, all measured in April 2019.
#
#
Process Measures: Clinical Volume and Adoption of Intervention Strategies
Clinical Volume
The mean (SD) number of patient encounters per half-day clinic for the observed ophthalmologists was 30.9 (4.6) during preimplementation, 29.4 (2.0) during early postimplementation, and 28.7 (3.4) at 6 months postimplementation. EHR implementation was not associated with any significant changes in clinical volume (p = 0.37).
#
Single Sign-On Functionality
SSO capabilities were successfully installed on all desktop workstations in the ophthalmology department before implementation. All ophthalmologists observed used SSO; none typed in their log-in credentials after SSO became active. Ophthalmologists initiated face-to-face communication with patients while tapping their badge to the proximity card reader, which automatically logged into the workstation and opened the EHR to that patient's chart. This eliminated the need to first face the computer to type in log-in credentials on the desktop, click the EHR shortcut icon, type in EHR log-in credentials, load the clinic schedule, and click to open the patient's chart. With SSO, log-in proceeded in the background, enabling ophthalmologists to direct attention to the patient first when entering the room.
#
Use of EHR in Mobile and Tablet Applications
Of the four observed ophthalmologists, two (50%) activated functionality to enter orders, review allergies and medications, take clinical photographs, and select patient pharmacies from the mobile app. This allowed simultaneous EHR documentation on two different devices and improved efficiency. For example, ophthalmic technicians often assisted in taking clinical photographs of the patient using the tablet-based EHR app while the ophthalmologist documented the progress note. These pictures were directly deposited into the EHR using the tablet's camera and the EHR app. Previously, ophthalmic technicians took photographs using separate digital cameras, which then had to be separately uploaded into the EHR and required a substantial amount of time and effort outside of clinic.
#
Adoption of Various EHR Features Based on the EHR Efficiency Portal
Over half (52%) of orders signed by observed ophthalmologists had team contributions, compared with 40% nationally ([Table 3]). The ophthalmologists in this study had more customized order lists (user-level preference list entries) compared with national averages. Nearly all (95%) visits were closed the same day as scheduled appointments.
#
Documentation Burden Based on the EHR Efficiency Portal
Novel workflows were developed to facilitate problem-based charting in an effort to reduce note length. In response to training about these workflows and various methods to facilitate rapid and succinct note creation, the average note length of the observed ophthalmologists was several hundred characters shorter than that of national averages when examining all notes (991 vs. 1,200) and progress notes specifically (929 vs. 1,400; [Table 3]) at 6 months postimplementation based on the EHR usage metrics provided by the vendor. In addition, the observed ophthalmologists had a higher average number of personalized SmartPhrase note composition templates (78 vs. 67). About 15% of notes were composed with copy/paste, compared with 31% nationally.
#
#
Balancing Measures: Patient Satisfaction
In total, 374 patients responded to Press Ganey surveys assessing patient satisfaction during the study period. In October 2018, the month immediately following EHR implementation, patient satisfaction scores were slightly decreased compared with the paper baseline ([Fig. 3]). For the remainder of the study period, an upward trend in patient satisfaction scores was observed, with a 20% increase in patient satisfaction by 6 months postimplementation ([Fig. 3]).
#
#
Discussion
Although EHRs can be leveraged for care coordination, population health surveillance, clinical research, and other beneficial applications, their use has also presented challenges. Physician burnout has been attributed at least in part to decreasing face-to-face communication in the EHR era. Here, we deployed a range of QI interventions during the implementation of an EHR in the outpatient ophthalmology setting to mitigate some of these unintended consequences. Our key findings were: (1) time spent exclusively talking face-to-face with patients returned to levels equivalent to paper charts; (2) increased documentation time associated with EHRs was less than previously reported; and (3) ophthalmologists adopted EHR personalization features and engaged in team-based orders with appropriate training and workflow development.
One of the primary concerns about widespread EHR use is its potential adverse effect on the physician–patient interaction.[12] [13] [14] The computer has been described as the “third wheel” in this interaction.[12] We recorded the ophthalmologists as “talking with the patient” only when talking was the sole activity. Although time spent exclusively talking with the patient decreased significantly in the early postimplementation period, by 6 months after implementation it returned to levels equivalent to paper charts. Given that talking was only recorded if no other activity was occurring simultaneously, these results show that ophthalmologists can engage in dedicated face-to-face communication at levels comparable to paper charts after a relatively short period of adaptation. Notably, patient satisfaction scores also demonstrated an upward trend throughout the study period, which further supports the finding that the observed ophthalmologists were able to maintain positive interactions with patients even while adapting to an electronic documentation system. Desktop computer use is associated with inherent changes in body position, eye contact, and communication patterns,[13] but expansion of EHRs into mobile- and tablet-based applications may lessen some of these changes (i.e., in body position). However, ophthalmologists in our study did not routinely use mobile- and tablet-based EHR applications, perhaps because these formats still offer less functionality than desktop versions. Designing more functionality into flexible EHR formats represents an area for future growth.
In this study, the mean total time to complete each encounter was 1.1 minutes longer with EHR use at 6 months postimplementation. Although this difference was statistically significant, it is strikingly shorter than that reported by Chiang et al at another academic ophthalmology department, where EHR took 6.8 minutes longer per encounter than paper based on observations from two ophthalmologists.[34] Thus, the difference in total encounter time between paper charts and EHR use may be less pronounced than previously thought. This was not simply reflecting changes in clinical volume, since there were no significant changes in clinical volume for the observed ophthalmologists when comparing study phases before and after EHR implementation. The total time increase here was also less than a previously reported EHR-to-EHR transition at an academic glaucoma practice encompassing 131 patient encounters, in which the average time spent by patients with their physician in clinic increased from 11 minutes before to 14 minutes 6 months after the EHR transition, which was statistically significant.[35] Based on comparisons with these prior studies in the ophthalmology literature, the QI strategies we used may have lessened the anticipated impact of the paper-to-EHR transition. Moreover, the strategies described in this study are all applicable to existing EHR users and therefore could potentially be used for EHR-to-EHR transitions as well.
Using data from the EHR efficiency portal at 6 months postimplementation, the QI strategies employed here were associated with shorter EHR use time, documentation efficiency, adoption of existing personalization features within the EHR, and implementation of team-based orders, as the observed ophthalmologists outperformed national averages on a range of metrics in these domains. Notably, the observed ophthalmologists did not appear to achieve face-to-face communication during the encounter simply by deferring note-writing and documentation completion to after-hours or subsequent days. Their after-hours workload metrics were well below national averages, and 95% of visits were closed the same day as the appointment. This corresponded with direct observations. In addition, these ophthalmologists were not achieving documentation efficiency by simply copying and pasting notes either. Their usage of that feature (15%) was less than half of the national average (31%).
The outcomes from both the manual time–motion observations and the EHR efficiency portal support the notion that optimization strategies can achieve success without major redesign of the EHR interface or employing scribes. Patient-centeredness was promoted by streamlining workflow processes (such as sign-on procedures, team-based orders, and problem-based charting) and leveraging existing EHR functionalities. A key factor was a high level of institutional investment, the importance of which was highlighted in a survey of >72,000 physicians across 150 provider organizations, wherein approximately 70% of the variance in physician satisfaction with EHRs was explained by initial training, trust in IT (or organizational culture), and the level of EHR personalization.[46]
Strengths of this study included the large number of observed patient encounters, inclusion of multiple subspecialties, and multiple follow-up periods. Timing outcomes were examined in detail during almost 650 encounters across the duration of the study, exceeding prior time–motion studies in ophthalmology related to EHR implementations and transitions.[34] [35] Generalizability was improved by including multiple ophthalmic subspecialties. Having multiple time points of observation characterized the evolution of time requirements rather than depicting a single snapshot.
Limitations included not being able to assess the effects of individual intervention strategies separately and the possible limited generalizability to settings besides academic medical centers, which may have more resources for QI efforts compared with smaller community-based practices, as well as possible limited generalizability to other specialties or ophthalmic subspecialties that were not included in the study population. Not all ophthalmologists in the department implemented the EHR during the study period. This presented a limitation in that not all ophthalmic subspecialties could be studied. In addition, those who transitioned to EHR may have been more avid users of technology with “early adopter” mindsets. Thus, this may represent a “best-case scenario” in terms of physician selection. Understanding how training or other QI strategies might be tailored to late adopters, or those who may be relatively more resistant to changes in the EHR, represents another area of future study. Moreover, future studies to investigate the efficacy of these strategies among existing EHR users, other specialties, and/or geographically disparate clinics could help assess the generalizability of these interventions. Finally, our study focused on time spent on documentation and other clinical activities, but we did not examine the quality of the documentation itself.
#
Conclusion
In conclusion, QI strategies can promote face-to-face communication with patients while using EHRs, even to levels on par with paper charts, without requiring redesigning the EHR interface itself, the assistance of scribes, or deferral of documentation work to after-hours. While EHR use did require more documentation time, the increase was less pronounced than previously reported. There were no significant decreases in clinical volume, and patient satisfaction increased after EHR implementation. Moreover, these strategies are widely applicable to current EHR users and are not limited to paper-to-EHR transitions. Because the EHR serves as the primary medium of delivery for health IT innovations, studying and optimizing EHR use will become increasingly important in the years to come.
#
Clinical Relevance Statement
Informatics-enabled interventions such as developing SSO capabilities, activating mobile- and tablet-based applications, training on EHR personalization features, creating workflows to facilitate team-based orders, and utilizing problem-based documentation can help decrease documentation burden while using EHRs. We demonstrated that these strategies helped maintain time spent on face-to-face communication to levels equivalent to paper charting in a multispecialty ophthalmology clinic. Investments in ongoing training and workflow development are crucial for promoting efficiency of EHR use, particularly in high-volume settings.
#
Multiple Choice Questions
-
The time spent on physical examination of patients is __________ with EHR use compared with paper charts.
-
Longer.
-
Shorter.
-
About the same.
Correct Answer: The correct answer is option c. Multiple studies have demonstrated that the time spent on physical examination of patients is about the same, regardless of the documentation method. However, time spent on documentation can be variable—while EHR use was initially thought to increase efficiency, several studies have shown that EHR use is more commonly associated with an increased amount of time spent on clinical documentation. Similarly, time spent on face-to-face communication is commonly perceived to decrease as more attention is directed to screens. However, informatics-enabled strategies can be undertaken to promote face-to-face communication.
-
-
Which of the following factors contributes the most to physician satisfaction with EHRs?
-
Prior background in computer science.
-
Level of EHR personalization.
-
Salary.
Correct Answer: The correct answer is option b. A large multi-institutional survey found that 70% of the variance in physician satisfaction with EHRs was explained by initial training, trust in IT (or organizational culture), and the level of EHR personalization. Strategies to promote EHR personalization were used in this study to decrease documentation burden and facilitate efficiency of EHR use.
-
#
#
Conflict of Interest
S.L.B., M.F.C., M.R.H., L.O.-M., and R.E.-K. were supported by grants from National Institutes of Health (T15LM011271, R00LM12238, P30EY10572, P30EY022589, and UL RR031980), the Heed Ophthalmic Foundation fellowship, and by unrestricted departmental grants from Research to Prevent Blindness (RPB). The funding organizations had no role in the design or conduct of this research. M.F.C. is an unpaid member of the Scientific Advisory Board for Clarity Medical Systems (Pleasanton, California, United States), a consultant for Novartis (Basel, Switzerland), and an initial member of InTeleretina, LLC (Honolulu, Hawaii, United States). A.E.H. is a medical specialist at Advanced Clinical (Deerfield, Illinois, United States) and is employed at Verily Life Sciences (South San Francisco, California, United States). The remaining authors do not have any disclosures.
Acknowledgments
The authors gratefully acknowledge the technical support of John Cerda, EHR leadership and go-live support from Dean Pham, and imaging support from Amy Radonich. Student observers who assisted with time–motion data collection were Jennie Xu, Citra Khalil, Ala'a Erras, Dennis Tran, Nguyen Nguyen, Neil E. Shah, Vivian Do, Alyssa Cadiz, Alyssa Haack, Sonali Bhanvadia, and Manreet Brar.
Protection of Human and Animal Subjects
The study adhered to the tenets of the Declaration of Helsinki and was reviewed and approved by the UCSD Institutional Review Board. Animal subjects were not included.
-
References
- 1 Shanafelt TD, Hasan O, Dyrbye LN. , et al. Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014. Mayo Clin Proc 2015; 90 (12) 1600-1613
- 2 Jha A, Iliff A, Chaoui A. , et al. A crisis in health care: a call to action on physician burnout. AHRQ Patient Safety Network. Available at: https://psnet.ahrq.gov/resources/resource/32662/A-Crisis-in-Health-Care-A-Call-to-Action-on-Physician-Burnout . Accessed July 19, 2019
- 3 Merritt Hawkins. 2018 Survey of America's physicians: practice patterns and perspectives. Available at: www.merritthawkins.com/news-and-insights/thought-leadership/survey/2018-survey-of-americas-physicians-practice-patterns-and-perspectives/ . Accessed July 19, 2019
- 4 Sinsky C, Colligan L, Li L. , et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med 2016; 165 (11) 753-760
- 5 Arndt BG, Beasley JW, Watkinson MD. , et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
- 6 Tai-Seale M, Olson CW, Li J. , et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017; 36 (04) 655-662
- 7 Wright AA, Katz IT. Beyond burnout - redesigning care to restore meaning and sanity for physicians. N Engl J Med 2018; 378 (04) 309-311
- 8 Collier R. Rethinking EHR interfaces to reduce click fatigue and physician burnout. CMAJ 2018; 190 (33) E994-E995
- 9 Robertson SL, Robinson MD, Reid A. Electronic health record effects on work-life balance and burnout within the I3 Population Collaborative. J Grad Med Educ 2017; 9 (04) 479-484
- 10 Medscape. Medscape National Physician Burnout & Depression Report 2018. Available at: www.medscape.com/slideshow/2018-lifestyle-burnout-depression-6009235 . Accessed February 11, 2019
- 11 Mitsch C, Huber P, Kriechbaum K. , et al. eHealth 2015 special issue: impact of electronic health records on the completeness of clinical documentation generated during diabetic retinopathy consultations. Appl Clin Inform 2015; 6 (03) 478-487
- 12 Saleem JJ, Flanagan ME, Russ AL. , et al. You and me and the computer makes three: variations in exam room use of the electronic health record. J Am Med Inform Assoc 2014; 21 (e1): e147-e151
- 13 Frankel R, Altschuler A, George S. , et al. Effects of exam-room computing on clinician-patient communication: a longitudinal qualitative study. J Gen Intern Med 2005; 20 (08) 677-682
- 14 Rizer MK, Sieck C, Lehman JS, Hefner JL, Huerta TR, McAlearney AS. Working with an electronic medical record in ambulatory care: a study of patient perceptions of intrusiveness. Perspect Health Inf Manag 2017; 14 (Winter): 1g
- 15 Ross JS. Making electronic health records work better for patient care. JAMA Intern Med 2016; 176 (04) 560-560
- 16 Shachak A, Reis S. The impact of electronic medical records on patient-doctor communication during consultation: a narrative literature review. J Eval Clin Pract 2009; 15 (04) 641-649
- 17 Ehrlich JR, Michelotti M, Blachley TS. , et al. A two-year longitudinal assessment of ophthalmologists' perceptions after implementing an electronic health record system. Appl Clin Inform 2016; 7 (04) 930-945
- 18 Guo U, Chen L, Mehta PH. Electronic health record innovations: helping physicians - one less click at a time. Health Inf Manag 2017; 46 (03) 140-144
- 19 Rizvi RF, Marquard JL, Seywerd MA. , et al. Usability evaluation of an EHR's clinical notes interface from the perspective of attending and resident physicians: an exploratory study. Stud Health Technol Inform 2017; 245: 1128-1132
- 20 Zahabi M, Kaber DB, Swangnetr M. Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors 2015; 57 (05) 805-834
- 21 Turchin A, Sawarkar A, Dementieva YA, Breydo E, Ramelson H. Effect of EHR user interface changes on internal prescription discrepancies. Appl Clin Inform 2014; 5 (03) 708-720
- 22 Bosse C, Kelly K. Improving EHR usability using LEAN methodology. Stud Health Technol Inform 2016; 225: 870
- 23 Hultman G, Marquard J, Arsoniadis E. , et al. Usability testing of two ambulatory EHR navigators. Appl Clin Inform 2016; 7 (02) 502-515
- 24 Gidwani R, Nguyen C, Kofoed A. , et al. Impact of scribes on physician satisfaction, patient satisfaction, and charting efficiency: a randomized controlled trial. Ann Fam Med 2017; 15 (05) 427-433
- 25 Shultz CG, Holmstrom HL. The use of medical scribes in health care settings: a systematic review and future directions. J Am Board Fam Med 2015; 28 (03) 371-381
- 26 Golob Jr JF, Como JJ, Claridge JA. Trauma surgeons save lives-scribes save trauma surgeons!. Am Surg 2018; 84 (01) 144-148
- 27 Mishra P, Kiang JC, Grant RW. Association of medical scribes in primary care with physician workflow and patient experience. JAMA Intern Med 2018; 178 (11) 1467-1472
- 28 Zallman L, Finnegan K, Roll D, Todaro M, Oneiz R, Sayah A. Impact of medical scribes in primary care on productivity, face-to-face time, and patient comfort. J Am Board Fam Med 2018; 31 (04) 612-619
- 29 Cowan TL, Dunlop WA, Ben-Meir M. , et al. Emergency consultants value medical scribes and most prefer to work with them, a few would rather not: a qualitative Australian study. Emerg Med J 2018; 35 (01) 12-17
- 30 Pranaat R, Mohan V, O'Reilly M. , et al. Use of simulation based on an electronic health records environment to evaluate the structure and accuracy of notes generated by medical scribes: proof-of-concept study. JMIR Med Inform 2017; 5 (03) e30
- 31 Hribar MR, Read-Brown S, Goldstein IH. , et al. Secondary use of electronic health record data for clinical workflow analysis. J Am Med Inform Assoc 2018; 25 (01) 40-46
- 32 Baxter SL, Gali HE, Huang AE. , et al. Time requirements of paper-based clinical workflows and after-hours documentation in a multi-specialty academic ophthalmology practice. Am J Ophthalmol 2019; ;(March): S0002-9394(19)30120-5
- 33 Read-Brown S, Hribar MR, Reznick LG. , et al. Time requirements for electronic health record use in an academic ophthalmology center. JAMA Ophthalmol 2017; 135 (11) 1250-1257
- 34 Chiang MF, Read-Brown S, Tu DC. , et al. Evaluation of electronic health record implementation in ophthalmology at an Academic medical center (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc 2013; 111: 70-92
- 35 Pandit RR, Boland MV. The impact of an electronic health record transition on a glaucoma subspecialty practice. Ophthalmology 2013; 120 (04) 753-760
- 36 Lim MC, Boland MV, McCannel CA. , et al. Adoption of electronic health records and perceptions of financial and clinical outcomes among ophthalmologists in the United States. JAMA Ophthalmol 2018; 136 (02) 164-170
- 37 Sieja A, Markley K, Pell J. , et al. Optimization sprints: improving clinician satisfaction and teamwork by rapidly reducing electronic health record burden. Mayo Clin Proc 2019; 94 (05) 793-802
- 38 Longhurst CA, Davis T, Maneker A. , et al; Arch Collaborative. Local investment in training drives electronic health record user satisfaction. Appl Clin Inform 2019; 10 (02) 331-335
- 39 Stevens LA, DiAngi YT, Schremp JD. , et al. Designing an individualized EHR learning plan for providers. Appl Clin Inform 2017; 8 (03) 924-935
- 40 Downing NL, Bates DW, Longhurst CA. Physician burnout in the electronic health record era: are we ignoring the real cause?. Ann Intern Med 2018; 169 (01) 50-51
- 41 Baumann LA, Baker J, Elshaug AG. The impact of electronic health record systems on clinical documentation times: a systematic review. Health Policy 2018; 122 (08) 827-836
- 42 Poissant L, Pereira J, Tamblyn R, Kawasumi Y. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 2005; 12 (05) 505-516
- 43 Institute for Healthcare Improvement. Science of improvement: establishing measures. Available at: http://www.ihi.org:80/resources/Pages/HowtoImprove/ScienceofImprovementEstablishingMeasures.aspx . Accessed December 16, 2019
- 44 Press Ganey. Patient experience - patient-centered care. Available at: https://www.pressganey.com/solutions/patient-experience . Accessed July 26, 2019
- 45 R Core Team. R: a language and environment for statistical computing. 2013 . Available at: http://www.R-project.org/ . Accessed January 8, 2020
- 46 KLAS. Making sense of EHR satisfaction - KLAS blog. Available at: https://klasresearch.com/resources/blogs/2018/09/12/making-sense-of-ehr-satisfaction . Accessed May 29, 2019
Address for correspondence
-
References
- 1 Shanafelt TD, Hasan O, Dyrbye LN. , et al. Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014. Mayo Clin Proc 2015; 90 (12) 1600-1613
- 2 Jha A, Iliff A, Chaoui A. , et al. A crisis in health care: a call to action on physician burnout. AHRQ Patient Safety Network. Available at: https://psnet.ahrq.gov/resources/resource/32662/A-Crisis-in-Health-Care-A-Call-to-Action-on-Physician-Burnout . Accessed July 19, 2019
- 3 Merritt Hawkins. 2018 Survey of America's physicians: practice patterns and perspectives. Available at: www.merritthawkins.com/news-and-insights/thought-leadership/survey/2018-survey-of-americas-physicians-practice-patterns-and-perspectives/ . Accessed July 19, 2019
- 4 Sinsky C, Colligan L, Li L. , et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med 2016; 165 (11) 753-760
- 5 Arndt BG, Beasley JW, Watkinson MD. , et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
- 6 Tai-Seale M, Olson CW, Li J. , et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017; 36 (04) 655-662
- 7 Wright AA, Katz IT. Beyond burnout - redesigning care to restore meaning and sanity for physicians. N Engl J Med 2018; 378 (04) 309-311
- 8 Collier R. Rethinking EHR interfaces to reduce click fatigue and physician burnout. CMAJ 2018; 190 (33) E994-E995
- 9 Robertson SL, Robinson MD, Reid A. Electronic health record effects on work-life balance and burnout within the I3 Population Collaborative. J Grad Med Educ 2017; 9 (04) 479-484
- 10 Medscape. Medscape National Physician Burnout & Depression Report 2018. Available at: www.medscape.com/slideshow/2018-lifestyle-burnout-depression-6009235 . Accessed February 11, 2019
- 11 Mitsch C, Huber P, Kriechbaum K. , et al. eHealth 2015 special issue: impact of electronic health records on the completeness of clinical documentation generated during diabetic retinopathy consultations. Appl Clin Inform 2015; 6 (03) 478-487
- 12 Saleem JJ, Flanagan ME, Russ AL. , et al. You and me and the computer makes three: variations in exam room use of the electronic health record. J Am Med Inform Assoc 2014; 21 (e1): e147-e151
- 13 Frankel R, Altschuler A, George S. , et al. Effects of exam-room computing on clinician-patient communication: a longitudinal qualitative study. J Gen Intern Med 2005; 20 (08) 677-682
- 14 Rizer MK, Sieck C, Lehman JS, Hefner JL, Huerta TR, McAlearney AS. Working with an electronic medical record in ambulatory care: a study of patient perceptions of intrusiveness. Perspect Health Inf Manag 2017; 14 (Winter): 1g
- 15 Ross JS. Making electronic health records work better for patient care. JAMA Intern Med 2016; 176 (04) 560-560
- 16 Shachak A, Reis S. The impact of electronic medical records on patient-doctor communication during consultation: a narrative literature review. J Eval Clin Pract 2009; 15 (04) 641-649
- 17 Ehrlich JR, Michelotti M, Blachley TS. , et al. A two-year longitudinal assessment of ophthalmologists' perceptions after implementing an electronic health record system. Appl Clin Inform 2016; 7 (04) 930-945
- 18 Guo U, Chen L, Mehta PH. Electronic health record innovations: helping physicians - one less click at a time. Health Inf Manag 2017; 46 (03) 140-144
- 19 Rizvi RF, Marquard JL, Seywerd MA. , et al. Usability evaluation of an EHR's clinical notes interface from the perspective of attending and resident physicians: an exploratory study. Stud Health Technol Inform 2017; 245: 1128-1132
- 20 Zahabi M, Kaber DB, Swangnetr M. Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors 2015; 57 (05) 805-834
- 21 Turchin A, Sawarkar A, Dementieva YA, Breydo E, Ramelson H. Effect of EHR user interface changes on internal prescription discrepancies. Appl Clin Inform 2014; 5 (03) 708-720
- 22 Bosse C, Kelly K. Improving EHR usability using LEAN methodology. Stud Health Technol Inform 2016; 225: 870
- 23 Hultman G, Marquard J, Arsoniadis E. , et al. Usability testing of two ambulatory EHR navigators. Appl Clin Inform 2016; 7 (02) 502-515
- 24 Gidwani R, Nguyen C, Kofoed A. , et al. Impact of scribes on physician satisfaction, patient satisfaction, and charting efficiency: a randomized controlled trial. Ann Fam Med 2017; 15 (05) 427-433
- 25 Shultz CG, Holmstrom HL. The use of medical scribes in health care settings: a systematic review and future directions. J Am Board Fam Med 2015; 28 (03) 371-381
- 26 Golob Jr JF, Como JJ, Claridge JA. Trauma surgeons save lives-scribes save trauma surgeons!. Am Surg 2018; 84 (01) 144-148
- 27 Mishra P, Kiang JC, Grant RW. Association of medical scribes in primary care with physician workflow and patient experience. JAMA Intern Med 2018; 178 (11) 1467-1472
- 28 Zallman L, Finnegan K, Roll D, Todaro M, Oneiz R, Sayah A. Impact of medical scribes in primary care on productivity, face-to-face time, and patient comfort. J Am Board Fam Med 2018; 31 (04) 612-619
- 29 Cowan TL, Dunlop WA, Ben-Meir M. , et al. Emergency consultants value medical scribes and most prefer to work with them, a few would rather not: a qualitative Australian study. Emerg Med J 2018; 35 (01) 12-17
- 30 Pranaat R, Mohan V, O'Reilly M. , et al. Use of simulation based on an electronic health records environment to evaluate the structure and accuracy of notes generated by medical scribes: proof-of-concept study. JMIR Med Inform 2017; 5 (03) e30
- 31 Hribar MR, Read-Brown S, Goldstein IH. , et al. Secondary use of electronic health record data for clinical workflow analysis. J Am Med Inform Assoc 2018; 25 (01) 40-46
- 32 Baxter SL, Gali HE, Huang AE. , et al. Time requirements of paper-based clinical workflows and after-hours documentation in a multi-specialty academic ophthalmology practice. Am J Ophthalmol 2019; ;(March): S0002-9394(19)30120-5
- 33 Read-Brown S, Hribar MR, Reznick LG. , et al. Time requirements for electronic health record use in an academic ophthalmology center. JAMA Ophthalmol 2017; 135 (11) 1250-1257
- 34 Chiang MF, Read-Brown S, Tu DC. , et al. Evaluation of electronic health record implementation in ophthalmology at an Academic medical center (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc 2013; 111: 70-92
- 35 Pandit RR, Boland MV. The impact of an electronic health record transition on a glaucoma subspecialty practice. Ophthalmology 2013; 120 (04) 753-760
- 36 Lim MC, Boland MV, McCannel CA. , et al. Adoption of electronic health records and perceptions of financial and clinical outcomes among ophthalmologists in the United States. JAMA Ophthalmol 2018; 136 (02) 164-170
- 37 Sieja A, Markley K, Pell J. , et al. Optimization sprints: improving clinician satisfaction and teamwork by rapidly reducing electronic health record burden. Mayo Clin Proc 2019; 94 (05) 793-802
- 38 Longhurst CA, Davis T, Maneker A. , et al; Arch Collaborative. Local investment in training drives electronic health record user satisfaction. Appl Clin Inform 2019; 10 (02) 331-335
- 39 Stevens LA, DiAngi YT, Schremp JD. , et al. Designing an individualized EHR learning plan for providers. Appl Clin Inform 2017; 8 (03) 924-935
- 40 Downing NL, Bates DW, Longhurst CA. Physician burnout in the electronic health record era: are we ignoring the real cause?. Ann Intern Med 2018; 169 (01) 50-51
- 41 Baumann LA, Baker J, Elshaug AG. The impact of electronic health record systems on clinical documentation times: a systematic review. Health Policy 2018; 122 (08) 827-836
- 42 Poissant L, Pereira J, Tamblyn R, Kawasumi Y. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 2005; 12 (05) 505-516
- 43 Institute for Healthcare Improvement. Science of improvement: establishing measures. Available at: http://www.ihi.org:80/resources/Pages/HowtoImprove/ScienceofImprovementEstablishingMeasures.aspx . Accessed December 16, 2019
- 44 Press Ganey. Patient experience - patient-centered care. Available at: https://www.pressganey.com/solutions/patient-experience . Accessed July 26, 2019
- 45 R Core Team. R: a language and environment for statistical computing. 2013 . Available at: http://www.R-project.org/ . Accessed January 8, 2020
- 46 KLAS. Making sense of EHR satisfaction - KLAS blog. Available at: https://klasresearch.com/resources/blogs/2018/09/12/making-sense-of-ehr-satisfaction . Accessed May 29, 2019