Measuring Electronic Health Record Use in the Pediatric ICU Using Audit-Logs and Screen Recordings

Abstract

Background Time spent in the electronic health record (EHR) has been identified as an important unit of measure for health care provider clinical activity. The lack of validation of audit-log based inpatient EHR time may have resulted in underuse of this data in studies focusing on inpatient patient outcomes, provider efficiency, provider satisfaction, etc. This has also led to a dearth of clinically relevant EHR usage metrics consistent with inpatient provider clinical activity.

Objective The aim of our study was to validate audit-log based EHR times using observed EHR-times extracted from screen recordings of EHR usage in the inpatient setting.

Methods This study was conducted in a 36-bed pediatric intensive care unit (PICU) at Lucile Packard Children's Hospital Stanford between June 11 and July 14, 2020. Attending physicians, fellow physicians, hospitalists, and advanced practice providers with ≥0.5 full-time equivalent (FTE) for the prior four consecutive weeks and at least one EHR session recording were included in the study. Citrix session recording player was used to retrospectively review EHR session recordings that were captured as the provider interacted with the EHR.

Results EHR use patterns varied by provider type. Audit-log based total EHR time correlated strongly with both observed total EHR time (r = 0.98, p < 0.001) and observed active EHR time (r = 0.95, p < 0.001). Each minute of audit-log based total EHR time corresponded to 0.95 (0.87–1.02) minutes of observed total EHR time and 0.75 (0.67–0.83) minutes of observed active EHR time. Results were similar when stratified by provider role.

Conclusion Our study found inpatient audit-log based EHR time to correlate strongly with observed EHR time among pediatric critical care providers. These findings support the use of audit-log based EHR-time as a surrogate measure for inpatient provider EHR use, providing an opportunity for researchers and other stakeholders to leverage EHR audit-log data in measuring clinical activity and tracking outcomes of workflow improvement efforts longitudinally and across provider groups.

Keywords

EHRs and systems - EHR time - pediatrics - audit-log - databases - data validation and verification - intensive and critical care - workflow - physician

Background and Significance

In 2009, the Health Information Technology for Economic and Clinical Health Act promoted the adoption and meaningful use of Health Information Technology.[1] [2] The subsequent widespread adoption of electronic health records (EHRs) in the ambulatory (86%) and inpatient (96%) settings has resulted in a massive revamp of the day-to-day life of health care providers across the United States.[3] [4] EHRs are required to maintain audit-logs for tracking health information technology use.[5] [6] Also, referenced as access logs, EHR data, metadata, and/or timestamps, audit logs contain activity records of EHR users including login and logout timestamps that may be used to calculate measures of EHR usage such as task counts, durations and metrics such has work outside work, teamwork for orders, etc.[7] [8] While the audit-log data were originally designed to support auditing of inappropriate record access, it provides an opportunity to study clinical activities unobtrusively and at scale.[7] [8] [9]

Time spent in the EHR, specifically, has been identified as an important unit of measure of health care provider clinical activity as it is a commodity in limited supply for health care professionals and has been implicated as a risk factor for health care provider burnout.[7] [9] [10] [11] Audit-log data have shown promise in measuring time spent on ambulatory clinical activities,[12] [13] [14] [15] [16] but it is not known if EHR times derived from raw audit-log data will reliably correspond to inpatient clinical inactivity given differences in EHR use patterns between inpatient and ambulatory providers. As an example, ambulatory providers may tailor their EHR work around scheduled appointments. Any EHR usage outside these scheduled appointments may be considered work outside work in the ambulatory setting. However, similar appointment slots do not exist in the inpatient setting, which may lead to more fragmented EHR use through the day that need to be analyzed using different definitions of clinically relevant EHR time and perhaps different thresholds of EHR inactivity.

Audit-log data have been cross-referenced against external metrics of clinic and provider efficiency (e.g., clinic volume and patient wait times) and provider satisfaction (e.g., satisfaction with workload in the EHR, and satisfaction with amount of time in EHR after clinic hours) in the ambulatory setting to provide actionable information that may drive improvement.[11] [13] [17] [18] While researchers have utilized audit-log data to characterize EHR usage across different inpatient clinical rotations,[19] the lack of validation of audit-log based EHR times in the inpatient setting may have resulted in underuse of this data in studies focusing on provider efficiency, workflow optimization, etc. in the inpatient setting. There is currently a dearth of clinically relevant EHR usage metrics based on definitions consistent with inpatient provider clinical activity.[7] Moreover, the lack of validation of inpatient audit-log based EHR times may also contribute to a reliance on proprietary EHR use metrics developed by EHR vendors using definitions of EHR use that may not be consistent with inpatient clinical application.[20] Furthermore, the use of vendor-reported measures of EHR use hinders efforts to replicate, generalize, and compare research related to EHR usage, such as estimating time in clinical workflows and understanding EHR usability and provider satisfaction across EHR vendors and institutions.

Ethnographical observation, including in-person observation, audio recordings and interviews, and perception solicitation using surveys, have remained the most common forms of validating EHR log data.[21] [22] [23] However, with the advancement of technological innovation in health care environments, computational ethnographical data, including screen activities, eye tracking, motion capture, and real time locating systems (RTLS), are being increasingly harnessed to validate audit-log data.[24] Studies utilizing these traditional and computational ethnographical methodologies have validated ambulatory audit-log data.[12] [13] [14] [15] [16] Validation of inpatient audit-log data, however, has not been performed. Given these observations, EHR times derived from audit-logs available within EHR databases must undergo validation as a reliable measure of clinically relevant inpatient provider EHR use.

Objective

The objective of our study is to validate inpatient EHR times derived from audit-logs using observed times extracted from screen recordings of EHR usage.

Methods

Study Population

This study was conducted in a 36-bed pediatric intensive care unit (PICU) at Lucile Packard Children's Hospital Stanford, affiliated with Stanford University School of Medicine, between June 11 and July 14, 2020. Attending physicians, fellow physicians, hospitalists, and advanced practice providers with ≥0.5 full-time equivalent (FTE) for the prior four consecutive weeks and at least one EHR session recording were included in the study.

Electronic Health Record Audit-Logs

Epic (Epic Systems, Verona, Wisconsin, United States) is the EHR system used at our institution. Epic Hyperspace is the application client hosted by Citrix Systems (Citrix Systems Inc., Santa Clara, California, United States) and is presented to all PICU providers. Providers have the option to register for access to Epic through Haiku or Canto, the mobile- and tablet-optimized interfaces to Epic, respectively. PICU providers can access the EHR at sit-down workstations or computer-on-wheels (e.g., during morning rounds) while in-hospital and remotely on laptops and personal computers. EHR audit-logs are recorded in the EHR's database (Clarity) and contain login and logout timestamp data for EHR sessions. An EHR session is defined as a period of EHR usage flanked by login and logout events.

EHR audit-log data were reviewed for each PICU provider-day. Total audit-log EHR-time per provider per day in minutes was calculated from log-in and log-out times recorded in the EHR's database. If a provider were to login to the EHR on a different workstation while still remaining logged into the EHR on a different workstation, the prior login session time recording would be terminated on a standard basis to avoid overestimation of login time. We further calculated active EHR-time from audit logs using a threshold of inactivity of 1 minute,[19] (e.g., a provider accessed an EHR activity at timestamp A followed by a different activity at timestamp B. Considering a threshold of inactivity of 1 minute, if the time interval between timestamp A and timestamp B exceeded 1 minute, 1 minute was attributed to the first activity and idleness was assumed for the rest of time interval between timestamp A and timestamp B). EHR activities in audit logs were divided into eight categories as follows: chart review, chart updating, note review, note entry, order entry, order review, result review, and other (including activities such as printing reports, creating new note templates, use of smart tools, etc.).

Electronic Health Record Session Recordings

For our study, the Citrix session recording player (1999–2021 Citrix Systems, Inc) was used to retrospectively review screen recordings that were captured as the provider interacted with the EHR. EHR sessions completed within Hyperspace are recorded and stored for a duration of 14 days by the Citrix Session Recording Player, primarily for use by the Department of Information Services teams for technical support and administrative purposes. One research team member collected log-in time, log-out time, and inactive time, as defined below, from these video recordings through manual observation. A second research team member then reviewed 20% of randomly chosen recordings independently to evaluate inter-rater reliability given the manual nature of data abstraction.

Definitions of Observed Electronic Health Record Usage Measures

Log-in time was defined as the time at which a provider reached the landing page of the EHR workspace after successfully signing in using one's credentials. Log-out time was defined as the time at which the provider is no longer in the EHR workspace. This could be a result of the provider manually logging off or a lockout forced by the system after a period of inactivity for security reasons. The period of inactivity that triggers a lockout is dependent on the type of workstation being used by the provider (i.e., single-user or dedicated versus multiuser or shared) and ranged between 5 and 15 minutes. For EHR sessions where providers were logged out due to a lockout after a period of inactivity, the timestamps at the start of inactivity (i.e., no mouse or keyboard activity for the entire duration) and appearance of the lockout screen were noted. Inactive time was defined as the duration between the time at which a provider became inactive until the lockout screen appeared. Total, active and inactive observed EHR-times per provider per calendar day ([00:00:00–23:59:59], referred to as provider days in this manuscript) were calculated from the observed timestamps at log-in, log-out, start of inactivity, and appearance of lock-out screen.

Pediatric Intensive Care Unit Clinical Shifts

Available PICU clinical shift data for the duration of the study were extracted from Amion (1999–2021 Spiral Software), an online scheduling platform. Work hours per calendar day were defined as clinical hours covered while on shift each 24-hour period (00:00:00–23:59:59).

Data Linkage

Each provider, provider group, and provider-day were assigned unique IDs to link audit-log, observed EHR-time data, and clinical shift data.

Statistical Methods

We determined that with data from 82 unique provider-days, we would be able to detect a correlation (r ≥ 0.35) between the audit-log based EHR-time and observed EHR-time data with 90% power at an α of 0.05. Data are represented as medians, interquartile ranges, and ranges. Correlations between the audit-log based EHR-time and the observed EHR-time are described with Pearson product-moment correlation coefficients by using Fisher's transformation to generate 95% confidence intervals. To quantify the relationship between audit-log based EHR-time and observed EHR-time, we used linear regressions with cluster-robust standard errors to account for clustering within providers. The difference between audit-log based and observed EHR-times were described by using raw minutes and percentages. Statistical analyses were performed by using R statistical software (R Foundation for Statistical Computing, Vienna, Austria, Version 1.2.5042) and Stata 15.1 (StataCorp, College Station, Texas, United States). The Stanford Institutional Review Board approved this study with waiver of informed consent.

Results

Study Sample

Across 43 PICU providers identified, a total of 4,329 EHR sessions were recorded in the EHR's database between June 11 and July 14, 2020. These corresponded to data on 545 unique provider days, of which 202 were randomly chosen using a random number generator and reviewed for availability of EHR session recording player data. Session recordings for 94 unique provider days were not yet archived and were thus available for review. Three provider days had incomplete EHR session recordings likely due to purge of old data, owing to a large number of logins in the past 14 days. These provider days were excluded. Clinical shift data were collected for the remaining 91 provider days from the scheduling platform.

Electronic Health Record Times Derived from Audit Logs

Audit-logs recorded a median total EHR time of 61.0 (IQR: 25.0–150.5, range = 2.0–416.0) minutes per provider day for all providers (including providers days not coinciding with clinical shifts). Median audit-log based total EHR-time for all attendings per provider-day was 51 (IQR: 25.5–101.25, range = 5–283) minutes, for all fellows per provider day was 139.5 (IQR: 16–194.8, range = 2–301) and for all advanced practice provider (APPs)/hospitalists per provider day was 250 (IQR: 36.5–312, range = 18–416) minutes ([Table 1]). The top three EHR activity types all providers spent time on were chart review (median = 10.5 [IQR: 0–19.7, range = 0–48] minutes), other (median = 7.5 [IQR: 0–15.7, range = 0–54] minutes), and note review (median = 4 [IQR: 0–8.7, range = 0–42] minutes).

Table 1 Summary of median audit-log based and observed electronic health record time per provider day

	All providers[a]			Providers with clinical shifts
	Audit-log		Observed	Audit-log	Observed
Total EHR time (in minutes)	61.0 (IQR: 25.0–150.5, range = 2.0–416.0)		63.0 (IQR: 25.00–142.50, range = 2.0–417.0)	136 (IQR: 54.2- 198.75, range = 11–355)	129.42 (IQR: 59.29–199.03, range = 21.83–356.83)
Active EHR time (in minutes)	10 (IQR: 0–34.5, range = 0–121)		47.0 (IQR: 18.50–105.00, range = 0–377.00)	29.5 (IQR: 0–60.5, range = 0–121)	89.36 (IQR: 41.16–139.75, range = 14.62–289.70)
	All attendings[a]		All fellows[a]		All APPs/hospitalists[a]
	Audit-log	Observed	Audit-log	Observed	Audit-log	Observed
Total EHR time (in minutes)	51 (IQR: 25.5–101.25, range = 5–283)	51 (IQR: 27.5–100.75, range = 5–295)	139.5 (IQR: 16–194.8, range = 2–301)	151 (IQR:16–195.5, range = 2–280)	250 (IQR: 36.5–312, 18–416)	248 (IQR 34–314, range = 12–417)
Active EHR time (in minutes)	5 (IQR: 0–22.5, range = 0–75)	38.5 (IQR: 19–77.2, range = 0–214)	26 (IQR: 6–43, range = 0–77)	96.5 (IQR: 16–146.2, range = 2–272)	51 (IQR: 15–82, range = 3–121)	125 (IQR 32.5–277, range = 4–377)
	Attendings with clinical shifts		Fellows with clinical shifts		APPs/hospitalists with clinical shifts
	Audit-log	Observed	Audit-log	Observed	Audit-log	Observed
Total EHR time (in minutes)	99 (IQR: 42.7–144.2, range = 11–283)	99.5 (IQR: 48–138, range = 21–295)	196.5 (IQR: 167.8–278.5, range = 132–301)	198 (IQR: 175.2- 241.2, range = 127–280)	267.5 (IQR: 188.5–310.5, range = 47–355)	260.5 (IQR 185–310.5, range = 47–356)
Active EHR time (in minutes)	12 (0–36.5, range = 0–75)	72.5 (IQR: 32–104.5, range = 14–214)	49 (IQR: 34–67.7, range = 0–77)	150.5 (IQR: 100.2–181.2, range = 87–272)	82 (IQR: 54.5–97.5, range = 16–121)	149.5 (IQR 123.5–252.8, range = 47–289)

Abbreviations: EHR, electronic health records; IQR, interquartile range.

^a Includes provider-days not associated with clinical shifts.

Median active EHR-time per provider day for all providers derived from audit logs using a conservative 1 minute threshold of inactivity was 10 (IQR: 0–34.5, range = 0–121) minutes. Median active EHR time for all attendings per provider day was 5 (IQR: 0–22.5, range = 0–75) minutes, for all fellows per provider day was 26 (IQR: 6–43, range = 0–77) minutes and for all APPs/hospitalists per provider day was 51 (IQR: 15–82, range = 3–121) minutes.

Median total EHR-time for providers with clinical shifts per provider-day was 136 (IQR: 54.2– 198.75, range = 11–355) minutes. Median active EHR time per day for providers with clinical shifts was 29.5 (IQR: 0–60.5, range = 0–121) minutes, for attendings with clinical shifts was 12 (IQR: 0–36.5, range = 0–75) minutes, for fellows with clinical shifts was 49 (IQR: 34–67.7, range = 0–77) minutes and for APPs/hospitalists with clinical shifts was 82 (IQR: 54.5–97.5, range = 16–121) minutes.

Electronic Health Record Times Derived from Manual Review of Session Recordings

EHR sessions from 31 unique providers spanning 91 provider-days were observed. EHR use patterns varied by provider type ([Table 1] and [Fig. 1]). Providers logged in to the EHR a median of 2 (IQR: 1–5, range = 1–8) times per day. Median observed total EHR-time for all providers per provider-day was 63.0 (IQR: 25.00–142.50, range = 2.0–417.0) minutes. Median total EHR time for all attendings per provider-day was 51 (IQR: 27.5–100.75, range = 5–295) minutes, for all fellows per provider-day was 151 (IQR: 16–195.5, range = 2–280) minutes and for all APPs/ hospitalists per provider day was 248 (IQR: 34–314, range = 12–417) minutes. We observed strong inter-rater reliability between the two reviewers ([Fig. 2]).

Fig. 1 Observed total electronic health record use by time of day among pediatric intensive care unit providers.

Fig. 2 Inter-rater reliability between reviewers.

Median observed active EHR time for all providers per provider day was 47.0 (IQR: 18.50–105.00, range = 0–377.00) minutes. Median observed active EHR-time for all attendings per provider day was 38.5 (IQR: 19–77.2, range = 0–214) minutes, for fellows per provider-day was 96.5 (IQR: 16–146.2, range = 2–272) minutes and for APPs/hospitalists per provider day was 125 (IQR: 32.5–277, range = 4–377) minutes. Median observed inactive EHR time per provider day was 10 (IQR: 0–35.0, range = 0–150) minutes. Inactive time accounted for 24% of total observed EHR-time across all provider days, with a median of 18% (IQR: 0–38.00%, range = 0–84.62%) per provider day.

A total of 45 of 91 provider days had corresponding scheduled PICU shifts. These providers with corresponding PICU shifts were noted to have median observed total EHR time per provider day of 129.42 (IQR: 59.29–199.03, range = 21.83–356.83) minutes, median observed active EHR time per provider day of 89.36 (IQR: 41.16–139.75, range = 14.62–289.70) minutes and median observed inactive EHR time per provider day of 32.33 (IQR: 10.01–62.77, range = 0.00–150.10) minutes. Median observed active EHR time for attendings with clinical shifts per provider day was 72.5 (IQR: 32–104.5, range = 14–214) minutes, for fellows with clinical shift per provider day was 150.5 (IQR: 100.2–181.2, range = 87–272) minutes and for APPs/hospitalists with clinical shifts per provider day was 149.5 (IQR: 123.5–252.8, range = 47–289) minutes.

Comparison of Audit-Log Based EHR Times and Observed EHR Times

Audit-log based EHR times correlated strongly with total observed logged-in time (r = 0.98, p < 0.001) and observed active EHR time (r = 0.95, p < 0.001; [Table 2]). Each minute of audit-log based total EHR time corresponded to 0.95 (0.87–1.02) minutes of observed total EHR time and 0.75 (0.67–0.83) minutes of observed active EHR time. Results were similar when stratified by provider role. Audit-log daily total EHR times were within 15 and 30 minutes of total observed EHR times for 83.5 and 93.4% of observations, respectively ([Fig. 3]). We further noted a strong correlation (r = 0.72, p < 0.01) between audit-log based active EHR times (using a conservative threshold of inactivity of 1 minute[19]) and observed active EHR times (observed active EHR time = total observed EHR time–inactive time; inactive time was defined as the duration between the time at which a provider became inactive until the lockout screen appeared, median 10 [IQR: 0–35.0, range = 0–150] minutes per provider day). Each minute of audit-log based active EHR time corresponded to 1.9 minutes of observed active EHR time.

Fig. 3 Difference between audit-log based total EHR-time and observed total-EHR time. EHR, electronic health record.

Table 2 Difference between audit-log based total EHR time and observed total and active EHR time

		Observed total EHR time		Observed active EHR time
Provider type, n	Provide days	r (95% CI)	β-coefficient[a]	r (95% CI)	β-coefficient[b]
All providers, 31	91	0.98 (0.97–0.99)	0.95 (0.87–1.02)	0.95 (0.92–0.96)	0.75 (0.67–0.83)
Fellows, 9	16	0.98 (0.94–0.99)	0.93 (0.83–1.04)	0.91 (0.77–0.97)	0.71 (0.50–0.91)
Attendings, 17	64	0.96 (0.94–0.98)	0.88 (0.68–1.08)	0.93 (0.89–0.96)	0.68 (0.56–0.79)
APP/hospitalists, 5	11	1.00 (1.00–1.00)	1.01 (1.00–1.02)	0.96 (0.83–0.99)	0.84 (0.77–0.90)

Abbreviations: APP, advanced practice provider; CI, confidence interval; EHR, electronic health records.

^a Estimates via linear regressions with cluster-robust standard errors clustered at the level of individual provider, dependent variable is total observed logged-in time, and independent variable is audit log time.

^b Estimates via linear regressions with cluster-robust standard errors clustered at the level of individual provider, dependent variable is total observed active time, and independent variable is audit log time.

Discussion

In this cross-sectional observational analysis of audit-log versus observed EHR time among pediatric critical care providers, we observed that inpatient audit-log based EHR-times correlate strongly with both total observed EHR time and observed active EHR time. We noted strong correlations between audit-log based and observed EHR-times across all provider groups despite their varied EHR use patterns and workflows.

The findings of this inpatient study build upon prior EHR timestamp validation efforts in ambulatory settings. Consistency between EHR system event-log data and direct observation findings for the entire workday was reported in a study involving family medicine physicians in a single-system residency and community clinic-based setting using direct time-motion observation methods.[12] In studies examining workflow in ophthalmology clinics, EHR timestamp analysis produced encounter time estimates that were within 3 minutes of the observed times for 76.6 to 80% of the appointments.[13] [14] [15] In a 2017 study conducted in 48 primary care departments of a community-based health care system, it was noted that the audit log-based estimates of encounter times were two minutes shorter than those based on in-person observation and three minutes shorter than those based on audio recordings.[16] Of note, our study looks at EHR time durations for inpatient provider days, consisting of 12- to 24-hour shifts, as opposed to individual appointment windows, which may account for the larger absolute differences between audit-log based and observed EHR times as calculated here ([Fig. 3]).

Moreover, EHR use patterns for ambulatory and inpatient providers may have differences owing to their workflows. Inpatient providers in the PICU may be more likely to step away from their workstations for unanticipated or urgent patient care, or ad hoc conversations than their ambulatory colleagues, for example, going to bedside frequently to check patient status while entering medication orders into the patient's chart. These frequent interruptions may lead to inaccurate estimates of active and/or inactive times, depending on the specific thresholds used to define inactivity which can be as low as 1 second intervals without clicks and keyboard strokes as used by EHR vendors (5 seconds as used by Epic at our institution to calculate active EHR-use metrics in proprietary reports). In our study, while observing EHR sessions, we defined providers as actively interacting with the EHR if they returned to their prior EHR session and manually logged out prior to automatic lockout consistent with clinically relevant active EHR time. True inactive EHR time may be underestimated using our definition of inactive time by failing to capture inactive time not leading to a timed lockout. However, we noted a strong correlation between audit-log based active EHR-time (derived using a conservative threshold of inactivity of 1 minute) and observed active EHR time, suggesting that calculation of active EHR time from audit logs is feasible and may be consistent with clinically relevant active EHR time. Although, further studies are needed to detail clinical setting-based definitions of EHR inactivity that may result in a more accurate reflection of clinically relevant active EHR activity, the strong correlation between audit-log based total EHR time and observed total EHR time and observed active EHR-time makes total EHR times derived from routinely recorded audit-log data a useful surrogate for active EHR use in the inpatient setting. EHR times derived from audit logs may lend itself to creation of clinically relevant inpatient EHR usage metrics akin to core EHR use measures proposed by Sinsky et al that reflect various dimensions of practice efficiency in the ambulatory setting.[7] Our validation of inpatient audit-log based EHR time has immeasurable potential to begin to understand and address not yet explored topics such as provider wellbeing, work-life integration, EHR fatigue, provider efficiency, documentation/order entry errors, effectiveness of EHR training programs, etc. in the inpatient setting.

As the first study to our knowledge that validates inpatient EHR timestamp data, it is not yet known how generalizable these findings are to other inpatient settings, provider types, or EHR vendors. Our finding of strong correlation between audit-log based and observed EHR times for three very different provider types, that is, attendings, fellows, and APPs/hospitalists in the PICU may signal that these findings are robust to EHR usage patterns and may suggest that they may hold relevance to providers in other settings as well. There remains a need for replication of these findings in other inpatient settings using similar methodology. While our study validates audit-log based EHR times from the application client presented to PICU providers at our institution, it does not validate EHR times recorded through applications that enable mobile device access (i.e., Haiku or Canto, Epic Systems, Verona, Wisconsin, United States). This may have led to underestimation of the total amount of time spent by PICU providers on the EHR across all devices. Observed inactive EHR-time, as defined in this study, might have underestimated true inactive time by failing to capture inactive time not leading to a timed lockout as. By the same token, our definition of inactive EHR time may have overestimated active EHR time ([Table 1]). Neither EHR audit logs nor session recordings differentiated between clinical and nonclinical work in the EHR (i.e., quality improvement or other administrative work, chart review for research, etc.).

We noted that the session recording player serves as a valuable tool in remotely validating audit-log timestamp data with the advantage of observing providers asynchronously, therefore minimizing the Hawthorne effect which may bias data collected in-person. We also recognize that this methodology has its limitations when compared with in-person observation. Provider activities outside of EHR use (e.g., using an internet search engine simultaneously with the EHR, reviewing articles stored in computer memory, etc.) are not recorded through these EHR session recordings. Future studies that include such data, from sources such as RTLS, video cameras in work settings, etc. may help fill in this gap in understanding work outside of the EHR in inpatient settings, although could come with increased privacy concerns.

Conclusion

This study found inpatient EHR audit-log based EHR times to correlate strongly with observed total and active EHR times among pediatric critical care providers. These findings support the use of audit-log based EHR time as a surrogate measure for inpatient provider EHR usage. This provides an opportunity for researchers and other stakeholders to leverage EHR audit-log data in measuring clinical activity through development of clinically relevant EHR usage metrics utilizing definitions relevant to inpatient workflows. Furthermore, these EHR usage metrics when cross-referenced against measures of patient outcomes, practice efficiency and provider satisfaction may guide improvement efforts longitudinally and across provider groups and institutions.[25]

Clinical Relevance Statement

We found that inpatient audit-log based EHR time correlates strongly with observed EHR time among pediatric critical care providers. These findings support the use of audit-log based EHR-time as a surrogate measure for inpatient provider EHR use, providing an opportunity for researchers and other stakeholders to leverage EHR audit-log data in measuring clinical activity through careful development of clinically relevant EHR usage metrics utilizing definitions relevant to inpatient workflows. Further, these EHR usage metrics when cross-referenced against measures of patient outcomes, practice efficiency, and provider satisfaction may guide improvement efforts longitudinally and across provider groups and institutions.

Multiple Choice Questions

1. Our study establishes a strong correlation between audit-log based EHR times and observed EHR times among pediatric critical care providers. How were the observed logged-in times captured in our study?

   • In-person observation

   • Perception solicitation using surveys

   • Provider interviews

   • EHR session recordings

   Correct Answer: The correction answer is option d. EHR session recordings is the correct choice. For our study, the Citrix session recording player was used to retrospectively review screen recordings that were captured as the provider interacted with the EHR. EHR sessions completed within Hyperspace are recorded and stored for a duration of 14 days by the Citrix Session Recording Player, primarily for use by the Department of Information Services teams for technical support and administrative purposes. Data collected from these video recordings included log-in time, log-out time, and inactive EHR time. The session recording player serves as a valuable tool in remotely validating audit-log timestamp data with the advantage of observing providers asynchronously, therefore minimizing the Hawthorne effect which may bias data collected in-person

2. Our study establishes strong correlation between audit-log based EHR times and observed EHR times in the following PICU provider groups:

   • Attending physicians

   • Fellow physicians

   • APPs/hospitalists

   • All of the above

   Correct Answer: The correct answer is option d. We included attending physicians, fellow physicians, hospitalists, and advanced practice providers practicing in the PICU in our study. We noted strong correlations between observed and audit-log EHR timestamp data across these provider groups despite their varied EHR use patterns and workflows which may signal that these findings are robust to EHR usage patterns and may suggest that they may hold relevance to providers in other settings as well.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by Stanford Institutional Review Board.

• References

• 1 Adler-Milstein J, Jha AK. HITECH act drove large gains in hospital electronic health record adoption. Health Aff (Millwood) 2017; 36 (08) 1416-1422
• 2 Washington V, DeSalvo K, Mostashari F, Blumenthal D. The HITECH era and the path forward. N Engl J Med 2017; 377 (10) 904-906
• 3 Office of the National Coordinator for Health Information Technology. “Office-based physician electronic health record adoption,” health IT quick-stat #50. Accessed January 2019 at: dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php
• 4 Office of the National Coordinator for Health Information Technology. “Percent of hospitals, by type, that possess certified health IT,” health IT quick-stat #52. dashboard.healthit.gov/quickstats/pages/certified-electronic-health-record-technology-in-hospitals.php . Accessed September 2018
• 5 Standards for Health Information Technology to Protect Electronic Health Information Created, Maintained, and Exchange =d, 45 C.F.R. § 170.210. Accessed 2021 at: https://www.law.cornell.edu/cfr/text/45/170.2102015
• 6 ASTM E2147-Standard Specification for Audit and Disclosure Logs for Use in Health Information Systems. Available at: https://www.astm.org/Standards/ E2147.htm
• 7 Sinsky CA, Rule A, Cohen G. et al. Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020; 27 (04) 639-643
• 8 Rule A, Chiang MF, Hribar MR. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc 2020; 27 (03) 480-490
• 9 Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C. EHR audit logs: A new goldmine for health services research?. J Biomed Inform 2020; 101: 103343
• 10 Gardner RL, Cooper E, Haskell J. et al. Physician stress and burnout: the impact of health information technology. J Am Med Inform Assoc 2019; 26 (02) 106-114
• 11 DiAngi YT, Stevens LA, Halpern-Felsher B, Pageler NM, Lee TC. Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers' perceived control over their workload in the EHR. JAMIA Open 2019; 2 (02) 222-230
• 12 Arndt BG, Beasley JW, Watkinson MD. et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-426
• 13 Hribar MR, Read-Brown S, Goldstein IH. et al. Secondary use of electronic health record data for clinical workflow analysis. J Am Med Inform Assoc 2018; 25 (01) 40-46
• 14 Read-Brown S, Hribar MR, Reznick LG. et al. Time requirements for electronic health record use in an academic ophthalmology center. JAMA Ophthalmol 2017; 135 (11) 1250-1257
• 15 Hribar MR, Read-Brown S, Reznick L. et al. Secondary use of EHR timestamp data: validation and application for workflow optimization. AMIA Annu Symp Proc 2015; 2015: 1909-1917
• 16 Tai-Seale M, Olson CW, Li J. et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine. Health Aff (Millwood) 2017; 36 (04) 655-662
• 17 Hribar MR, Read-Brown S, Reznick L. et al. Secondary use of EHR timestamp data: validation and application for workflow optimization. AMIA Annu Symp Proc.. Accessed 2015 at: https://pubmed.ncbi.nlm.nih.gov/26958290/
• 18 Hribar MR, Huang AE, Goldstein IH. et al. Data-driven scheduling for improving patient efficiency in ophthalmology clinics. Ophthalmology 2019; 126 (03) 347-354
• 19 Wang JK, Ouyang D, Hom J, Chi J, Chen JH. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. PLoS ONE. 2019; 14 (02) e0205379
• 20 Baxter SL, Apathy NC, Cross DA, Sinsky C, Hribar MR. Measures of electronic health record use in outpatient settings across vendors. J Am Med Inform Assoc 2020
• 21 Unertl KM, Novak LL, Johnson KB, Lorenzi NM. Traversing the many paths of workflow research: developing a conceptual framework of workflow terminology through a systematic literature review. J Am Med Inform Assoc 2010; 17 (03) 265-273
• 22 Zheng K, Guo MH, Hanauer DA. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research. J Am Med Inform Assoc 2011; 18 (05) 704-710
• 23 Lopetegui M, Yen P-Y, Lai A, Jeffries J, Embi P, Payne P. Time motion studies in healthcare: what are we talking about?. J Biomed Inform 2014; 49: 292-299
• 24 Patel VL, Kannampallil TG, Kaufman DR. Cognitive Informatics for Biomedicine: Human Computer Interaction in Healthcare. Springe =r International Publishing; 2015. DOI: 10.1007/978-3-319-17272-9
• 25 National Research Network | Department of Medicine. Accessed May 28, 2020 at: https://medicine.ucsf.edu/center-clinical-informatics-and-improvement-research/national-research-network

Publication History

Received: 29 January 2021

Accepted: 28 June 2021

Article published online:
11 August 2021