Applying the RE-AIM Framework for the Evaluation of a Clinical Decision Support Tool for Pediatric Head Trauma: A Mixed-Methods Study

Abstract

Background The overuse of cranial computed tomography (CT) to diagnose potential traumatic brain injuries (TBIs) exposes children with minor blunt head trauma to unnecessary ionizing radiation. The Pediatric Emergency Care Applied Research Network and the Clinical Research on Emergency Services and Treatments Network implemented TBI prediction rules via electronic health record (EHR) clinical decision support (CDS) to decrease use of CTs in children with minor blunt head trauma.

Objective This article aims to facilitate implementation and dissemination of a CDS alert into emergency departments around the country.

Methods We evaluated the EHR CT CDS tool through a mixed-methods analysis of 38 audio-recorded interviews with health care stakeholders and quantitative data sources, using the Reach, Efficacy, Adoption, Implementation, and Maintenance framework.

Results Reach —The demographics of participants enrolled in the clinical trial were consistent with national estimates of TBI prevalence. Efficacy—There was a variable and modest reduction in CT rates for the 8,067 children with minor head trauma whose clinicians received CDS. Adoption —The EHR CT CDS tool was well matched with the organizational mission, values, and priorities of the implementation sites. Implementation— The most important predisposing factors for successful implementation were the presence of an approachable clinical champion at each site and belief that the tool was a relevant, reusable knowledge asset. Enabling factors included an effective integration within the clinical workflow, organizational investment in user training, and ease of use. Maintenance — Reinforcing factors for the EHR CT CDS tool included a close fit with the institutional culture, belief that it was useful for providers and families, and a good educational and informational tool. As such, the EHR CT CDS tool was maintained in clinical practice long after study completion.

Conclusion Data from this mixed-methods study complement findings from the efficacy trial and provide critical components for consideration prior to integration and subsequent dissemination of the EHR CT CDS tool.

Trial Registration NCT01453621, Registered September 27, 2011

Keywords

clinical decision support - clinical trial - blunt head trauma - traumatic brain injury - implementation - child

Background and Significance

The emergency department (ED) is the most common medical setting for the evaluation of children with minor blunt head trauma.[1] Between 2001 and 2010, ED visits for traumatic brain injury (TBI) among children aged 14 years and younger increased 40 to 50%.[2] Although population-based estimates of computed tomography (CT) use in children with minor blunt head trauma are lacking, recent data suggest continued overuse of CT for TBI.[3] [4] Overuse of CT scans unnecessarily exposes children to ionizing radiation with its known potential to induce lethal malignancies. Therefore, children with minor blunt head trauma represent an ideal at-risk population to target for CT minimization strategies.[3]

The Pediatric Emergency Care Applied Research Network (PECARN) developed, validated, and subsequently studied the implementation of two TBI prediction rules (one for children younger than 2 years, and the other for children 2–18 years old) to optimize clinician decision-making regarding the use of CT scans.[5] [6] Subsequently, the PECARN implementation study[6] was a nonrandomized multicenter clinical trial with concordant controls comparing CT use before and after the implementation of the PECARN TBI prediction rules. Thirteen EDs in the United States were included in the implementation study: five were academic EDs in PECARN and eight were community EDs in the Kaiser Permanente Clinical Research on Emergency Services and Treatments (CREST) Network.

For the PECARN implementation trial, an electronic health record (EHR)-based CT clinical decision support (CDS) tool was developed and implemented in the Epic EHR through Web or native CDS services.[7] [8] [9] Tham et al describe the customization of the CDS alert into clinical workflows at different institutions.[7]

The CDS rules provided risk estimates of clinically important TBI (i.e., death from TBI, TBI requiring neurosurgery, hospitalization ≥ 2 nights, hospitalization with a positive CT, or intubation for TBI > 24 hours) and recommendations regarding whether or not a CT was recommended based on the PECARN TBI prediction rules. To inform the design of the EHR CT CDS tool, we (the PECARN investigative team) previously completed focus group discussions with local health care providers and thought leaders and interviews with key stakeholders.[10] We also conducted workflow evaluations at participating sites, with the goal of understanding local practices and workflows, and maximizing the acceptance and usefulness of the CDS.[10] The focus groups, interviews, and observations identified early barriers to adoption and implementation.[10] The completion of the multicenter PECARN implementation trial offered the opportunity to follow-up with participating clinicians and thought leaders and understand the implementation of the EHR CT CDS tool in more depth.

To support future dissemination and translation, our aim was to summarize the quantitative efficacy findings of the multicenter PECARN implementation trial and integrate with additional contextual/qualitative data to identify and understand the reach, adoption, implementation, and maintenance of the EHR CT CDS tool.

Methods

Study Design

We used a mixed-methods design that summarized quantitative data sources, including those from the PECARN implementation trial, and integrated them with postimplementation semistructured interviews with a sample of key stakeholders. We used the Reach, Efficacy, Adoption, Implementation, and Maintenance (RE-AIM) framework, which provides a structure to organize the study of intervention implementations.[11] [12] Beyond e fficacy, the framework focuses on the: r each of the intervention to a representative proportion of the target population; a doption of the intervention across a broad and representative proportion of settings; i mplementation details; and m aintenance of the intervention post intervention.[11] [12] The RE-AIM framework has been customized to clinical informatics[13] by adding pertinent questions related to predisposing, enabling, and reinforcing factors.[14] Predisposing factors influence motivation to undertake a behavior, and enabling factors make it possible for individuals to change either behaviors or the environment.[15] [16] In the context of a clinical informatics intervention, this also includes institutional commitment and leadership support, integration of the EHR CT CDS tool into the organizational context and workflow, time allowed for learning, investment in the change process, and user training.[13] Reinforcing factors include the individual, setting, and organizational factors that are required to maintain use of the CT CDS tool.[13] With regards to setting, this includes the extent to which the CT CDS tool had become part of routine practice and was being used at least 6 months after study completion.[13]

Quantitative Data Sources and Qualitative Samples

We used the following quantitative sources of data in this analysis: (1) the existing publication from the PECARN implementation trial[6]; (2) a national report of TBIs published by the Centers for Disease Control and Prevention,[17] and (3) a report of the market penetration of the Epic EHR.[18] [19] For the qualitative assessments, we conducted semistructured interviews at two academic teaching hospital EDs in PECARN and four community EDs within the Kaiser Permanente system in Northern California that participated in the implementation trial.[6] The semistructured interview sample consisted of health care providers (attending faculty physicians, residents, advanced practice nurses, staff nurses, and nursing and medical ED managers), and key stakeholders in information technology (data analysts, Chief Medical Information Officer).

Procedures

[Table 1] outlines the data sources, procedures used to collect the data, and the applicable RE-AIM dimensions. For the primary qualitative data collection, we developed a postimplementation semistructured interview guide based on the adoption, implementation, and maintenance dimensions of the RE-AIM framework. This included eliciting predisposing, enabling, and reinforcing factors for the implementation and maintenance of the EHR CT CDS tool. Within each institution, we requested to interview a range a participants representing different health care provider and administrative roles and years of experience in the role. A purposive convenience sample was interviewed by two research team members with experience conducting qualitative interviews (S.B. and R.M.C.). Written or verbal informed consent was obtained at each site at the beginning of each interview, depending on the local Institutional Review Board requirements. We conducted most interviews (n = 37) in person and one by telephone until thematic saturation was reached. We offered participants $20 gift cards after their interviews. Interviews averaged 30 minutes in duration, and were digitally recorded, and professionally transcribed verbatim.

Table 1 Data sources for each dimension of RE-AIM

Data sources	RE-AIM dimension	Procedure
Centers for Disease Control and Prevention Report[17]	Reach	Extract age, gender, race, and ethnicity data from the report
HealthIT.gov[18] and Epic data sheet[19]	Reach	Obtain market share of Epic EHR to inform generalizability of CT CDS tool
Published data from the PECARN implementation trial[6]	Reach, Efficacy/Effectiveness	Extract age, gender, race, and ethnicity data; Summarize primary study results
Semistructured postimplementation trial interviews	Adoption, Implementation, Maintenance	Develop semistructured interview guide; Conduct 38 interviews at 6 study sites
Supplemental tables from PECARN implementation trial[6]	Adoption	Extract user-specific adoption information

Abbreviations: CDS, clinical decision support; CT, computed tomography; EHR, electronic health record; PECARN, Pediatric Emergency Care Applied Research Network; RE-AIM, Reach, Efficacy, Adoption, Implementation, and Maintenance.

Data Analysis

The authors analyzed the interview transcripts using a qualitative descriptive approach to identify common themes related to adoption, implementation, and maintenance of the EHR CT CDS tool across types of sites (academic and community EDs). The interview analysis was primarily deductive, based on the adoption, implementation, and maintenance dimensions of the RE-AIM framework, and then the themes were inductively generated to further characterize the data. The analysis included three discrete steps. First, two authors with training in qualitative methods independently read each transcript, and defined codes in a data dictionary (R.M.C. and S.B.). The data dictionary also included a priori codes based on the literature, previous work, and the interview guide. Second, one author applied nodes to all of the transcripts based on the coding dictionary. Third, the authors met to review, discuss, and arrive at consensus and ensure that the coded data fit the agreed upon definitions of the five RE-AIM dimensions. Themes that emerged from three or more interviews were identified as common.

To ensure rigor of the qualitative findings, two authors conducted the majority of interviews and a peer debriefing immediately after each interview to summarize key points in the interview; one author confirmed the content of the audio recordings to ensure accuracy. All authors reviewed the coding procedure to ensure dependability and credibility and the coders adhered to the coding procedure.

Results

Demographic Characteristics of the Semistructured Interview Sample

We interviewed 38 participants, of whom 21 (55%) were attending physicians with a mean of 13 years of posttraining clinical experience, 8 (21%) were staff nurses with a mean of 9 years of experience in the ED, 2 (5%) were nurse practitioners/physician assistants, 2 were residents (5%), and 5 (13%) were other key stakeholders including a Chief Information Officer, 2 data analysts, and 2 nurse managers. The majority of the sample was female (66%) and white (82%).

Reach

The primary assessment of the Reach dimension (and thus generalizability of the study trial) focused on comparing the absolute number, proportion, and representativeness of those who participated in the intervention to national estimates of TBIs. A total of 8,067 participants younger than 18 years were enrolled in the implementation clinical trial after the EHR CT CDS was implemented.[6] [Table 2] notes that the demographic characteristics of the enrolled participants were comparable with national estimates of TBI-related ED visits by age (0–19 years) and race according to the Centers for Disease Control and Prevention.[17]

Table 2 Comparison of TBI-related ED visits in PECARN implementation trial (0–17 years) and national estimates (0–19 years) (based on CDC)

	PECARN n = 8,067[6] (%)	CDC[a] (%)[17]
Race
American Indian or Alaskan Native	17 (0.2)	−
Asian	287 (3.6)	3
Black	1,558 (19.3)	12
More than one race	521 (6.5)	−
Native Hawaiian or Other Pacific Islander	44 (0.5)	−
Unknown	1,524 (18.9)	24
White	4,116 (51.0)	61
Ethnicity		Data not available
Non-Hispanic/Latino	6,677 (82.8)
Hispanic/Latino	851 (10.5)
Unknown	539 (6.7)
Gender
Male	5,080 (63.0)	63
Female	2,987 (37.0)	37

Abbreviations: CDC, Centers for Disease Control; ED, emergency department; PECARN, Pediatric Emergency Care Applied Research Network; TBI, traumatic brain injury.

^a Centers for Disease Control and Prevention data (2002–2006).[17]

For this study, the Reach dimension also included the market penetration of the Epic EHR since Epic was the vendor for the EHR CT CDS tool. According to HealthIT.gov, as of March 2015, Epic was the largest primary hospital-based EHR participating in the Centers for Medicare and Medicaid Services EHR Incentive Program. Epic was the third largest EHR vendor in the United States.[18] [19] According to a 2017 Epic fact sheet, Epic currently has 350 client health systems in the United States which provide health care services to over half of the U.S. population (totaling nearly 190 million patients).[20] These data suggest that the potential Reach of EHR CT CDS tool is substantial.

Efficacy

The Efficacy dimension of the RE-AIM framework examines the impact on important primary and secondary outcomes.[13] Complete details on the trial results are found in the PECARN implementation publication.[6] In summary, for all children with minor head trauma (when all sites combined) intervention sites had small decreases in CT use (1.7–6.2% across sites; adjusted odds ratio 0.72 [95% confidence interval [CI] 0.53–0.99]). There were also variable decreases in CT use at the control sites.[6] The potential unintended consequences of implementing the CT decision support tool appeared to be minimal—there was no increase in CT rates among those who were not at very low risk of clinically important TBI and no increase in the “miss” rate for clinically important TBIs (with no missed children with neurosurgeries).[6]

Adoption

The Adoption dimension of the RE-AIM framework addresses the absolute number, proportion, and representativeness of settings and clinicians who were willing to initiate the EHR CT CDS tool. Based on the PECARN CDS implementation trial,[6] provider-specific adoption at each site varied based on ED site-specific staffing, workflow, and study protocol implementation. Overall, at the Kaiser Permanente (community) sites, attending faculty members completed most of the data (89% across EDs) followed by nurses (10%) and residents (1%). At the PECARN EDs, there was much more variation among sites. Of the 16,635 participants enrolled in the intervention from PECARN EDs, 54% were completed by attending/fellows, 27% by nurses, 10% by residents, 7% by nurse practitioner/physicians assistants, and 2% by other health care providers.[6] [Table 3] presents data from the semistructured interviews, noting that adoption of the EHR CT CDS tool was perceived as high across the sites and was reinforced by a strong “match with mission” of the institutions in which it was implemented.

Table 3 Select quotations related to the adoption, implementation, and maintenance dimensions of the RE-AIM framework

RE-AIM dimension	Theme	Select quotations
Adoption: Absolute number, proportion, and representativeness of settings and intervention agents who are willing to initiate the PECARN EHR CT CDS tool
	Match with Mission	“And just keeping up on the literature, we're champions of this, bringing it up and making it part of our electronic medical records, it's helpful.” (Attending physician #27) “So I think as a department, as a health network, we're accepting of it and are comfortable with the using of it and entering the data and using it to our advantage.” (Attending physician #15)
Implementation: Extent and consistency to which a program is delivered across programs and settings as intended after it is implemented
Predisposing factors: Occur before a behavior and influence motivation to undertake a particular behavior (factors include knowledge, attitudes, beliefs, values, self-efficacy, behavioral intentions, and existing skills); can occur at both the individual or organizational level	Perception of relevance of CDS	“It provides you clinical support to provide better quality of care to the patient; in the case of the child, talking about risk and benefits of potential intracranial injury versus that of a CT scan.” (ED Director/Attending physician #26) “I think that for our documentation as a physician, I think that it's really important to have decision tools documented in the EMR. I think from a medical legal perspective, it's important. I try to put in decision rules and things of that nature. Things that help me calculate risk, I try to incorporate that into my note.” (Resident #9) “I think decision tools are great…. I think using decision tools is very helpful and making them fairly accessible with an easy-to-use interface and something that can give you a quick answer helps support your decision. I think they're very useful tools to have in your back pocket.” (Attending physician #15) “From a nursing standpoint, it's…honestly, I don't know that it's that beneficial to us. It's not going to change what I do, how I care for them, how I assess them. I think it's more beneficial from the physician's standpoint.” (RN #12) “I guess if there would have been nurse leader, a nurse manager that supported it along with [the physician champion] with some of that learning as we went through, you know, how could we use this to our benefit.” (Nurse manager #28) “But then they [pediatricians] kind of get trumped and overruled by the trauma team who is going to do what they want anyways.” (RN #17)
	Clinical champion	“You've got to have champions that will support and demonstrate why this is an incredible tool. And once you get a few onboard, everyone will see the benefits of it.” (Attending physician #26)
Enabling factors: Precede and support behavior; include institutional commitment and central leadership support, integration of system into organizational context, time to allow learning, investment in change process, and adequate user training	Integrated Workflow	“It's coming at the right time, to the right person, with the right information, using the right channel, and in the right situation.” (Attending physician #3) “It's probably in a good location from a resident workflow issue.” (Attending physician #16)
	User training	“Of course, education for us when we first started it…this is why we're doing it, this is what we need…some posters of things like that to kind of help us. It was helpful to have pictures of where the head trauma was because at first we weren't sure…does this part count? So, it was helpful to have that graph.” (RN #12)
	Ease of use	“The ease. It's incredibly easy to use.” (Resident #9) “The fact that it's integrated into our medical records system, our electronic medical records, it makes it very easy to use; you don't have to pull anything down. And probably the biggest thing that makes it easy to use is that it's integrated into our electronic medical system.” (Attending physician #27) “It was really pretty simple, pretty straightforward, especially with the navigator. I know it was really easy in that sense.” (RN #29) “I like it. I think it's utilized appropriately. I think most nurses fill it out. It's pretty simple, you know. Just the flow sheet, fill it out, and it prompts you to ask the appropriate questions. I do think it's helpful.” (RN #2) “It flows very easily in Epic…it's very similar to our workflow for other things, so the learning curve was no big deal.” (RN #1) “I think at first it was difficult to use because not all of the physicians were onboard in the limited use of the CT scans, and being a very collaborative practice, we always run cases by our colleagues, and some are a little more lax in their use of the CT scans, and some tended to use the CT scan more than others, so you were always doubting yourself whether this was the right way. So now, using the CT scan is easier and it certainly easier on you and helps you sleep at night and removes any doubt from your head that there was a bleed because you're never sure unless you get a CT scan, frankly. But the data are very reassuring and it helps the more numbers that we get and the more the data shows this is safe, so it makes it easier. But initially, it was not easy because you just weren't sure.” (Attending physician #27) “It doesn't always trigger and I've tried before to put in as a chief complaint and make it trigger. But I have a hard time doing it.” (Attending physician #6) “The only thing that I get frustrated with is when I want it to fire and I can't because the chief complaint when they first walked in doesn't match. And if I add a new complaint for some reason, it doesn't pop. I don't know why. ” (Attending physician #6)
Maintenance: Degree to which a program becomes routine and part of everyday culture and norms of an organization at both the setting and individual level
Reinforcing factors	Institutional culture	“I think we hit all the 5 rights of CDS on this one. It's coming at the right time, to the right person, with the right information, using the right channel, and in the right situation….There's been no interest in turning it off. In fact, we've now had a request to do something similar for abdominal pain. They want some kind of assessment tool. The same methodology we did and we're hearing from our pediatric surgeons they want to do something like that to reduce the use of abdominal CTs for abdominal pain and switch to doing more ultra sounds or nothing or observation. So, I think it's been a success.” (Attending physician #3)
	Benchmarking tool	“There was feedback to providers to say, hey, you're not a very good user of the clinical decision rule, so people got some direct feedback about what percentile they were in…you are using it 75 or more percent of the time, you're using about 50 percent of the time, you're using it less than 25 percent of the time…and that feedback, I think, was kind of useful…it kind of spurs, oh, I've got to remember to do that. Just like any other kinds of ongoing feedback to your clinical performance is important.” (Attending physician #23) “I would love to be able to do give a provider specific feedback about their adherence.” (Attending physician #32)
	Informational tool	“I think it would be a great, if not shared decision-making tool, more of an informing, why we're advising the way we are.” (Attending physician #21)
	Usefulness	“I've actually been pretty pleased with using it, and hope that there will be more decision aids and more charts and things that we can use to help with shared decision making.” (Attending physician #8) “So, for me to get the feedback, I don't necessarily need the clinical decisions support that comes along with it, but I do find that it is beneficial, especially with a masses [residents from a large training program] that come with the Emergency Department… I think it's very useful for them to guide their decision making process.” (Attending physician #34)
	Education tool	“I find it really helpful with the residents. I think it's almost a better teaching tool than a decision-making tool, in some senses, because I think a lot of us do it automatically in our head now. But I think from a teaching standpoint or in a place where they don't play with kids all of the time, I think it would be really, really helpful.” (Attending physician #6) “I think that the visual aid would be really great for families to understand, to completely understand, and bring it down into concrete terms.” (Attending physician #8) “I think it's most useful to me in being able to tell families that want to know that we have a tool that we look at and it helps us make a clinical decision about whether or not based on the child's symptoms in the physical exam, we need to do neuroimaging. And that seems to alleviate some parent's fears about whether or not they needed to have a CT scan done.” (Nurse practitioner #20) “I think when it helps is when they're educating the patient or the family.” (Nurse manager #28)

Abbreviations: CDS, clinical decision support; CT, computed tomography; ED, emergency department; EHR, electronic health record; EMR, electronic medical record; PECARN, Pediatric Emergency Care Applied Research Network; RE-AIM, Reach, Efficacy, Adoption, Implementation, and Maintenance.

Implementation

The Implementation dimension of RE-AIM at the setting level takes into account the sites' fidelity to the protocol, including consistency of the delivery as it was originally intended.[13] Consistency of the intervention delivery was achieved by standardizing triggers, CDS prediction rules, and clinician messages, while allowing intended practice variation across sites (e.g., who entered the items on the trauma tool).[10] The participants identified both predisposing and enabling factors that supported the implementation of the CDS.

Predisposing Factors

Based on the interviews, the most important predisposing factors for successful implementation were the presence of an approachable clinical champion at each site and a strong perception that the EHR CT CDS tool was relevant to clinical practice by supporting quality care to patients ([Table 3] and [Fig. 1]). Across all sites, a clinical champion was essential for the implementation of the tool. There were divergent perspectives between and within providers at the different sites as to the degree of perceived relevance of EHR CT CDS tool. If health care providers believed, in general, that CDS tools made a difference in the clinical decision-making, they reported being more likely to use the tool. At the sites where nurses were responsible for completing the CDS tool, they felt that it was relevant to their role, but if they were not responsible for completing the tool, they felt that it was irrelevant to their scope of responsibility and tasks.

Fig. 1 Study methods and qualitative themes.

Among many providers, the EHR CT CDS tool was perceived as being a reusable knowledge asset and relevant for the development of a future toolkit of CDS resources. On the other hand, some providers thought that the EHR CT CDS tool might not be relevant in a high-volume, high-severity trauma setting where clinical decisions have to be made immediately without the aid of CDS in the EHR. Providers also felt the CDS tool would be most relevant to children seen in general rather than pediatric EDs (where the perception is that clinical comfort with young children is lower and awareness of the harms of CT to developing brains was perceived to be higher [in general EDs] than in pediatric EDs).

Enabling Factors

There were several predominant enabling factors described by respondents ([Table 3] and [Fig. 1]) that supported the use of the EHR CT CDS tool including: (1) institutional investment in user training, (2) integration into the clinical workflow, and (3) ease of use. Early on in the implementation, each site invested in user training, which included a clinical champion (ED site physician investigator) and study champions from information technology who provided ongoing education to all staff. A standardized message was created centrally, provided to all sites, delivered through a Grand Rounds-type format, and circulated to all physician and nursing staff. Given the unique workflows at each site,[10] the presentation was also tailored to address unique site features (i.e., charting at the point-of-care or at a central workstation). The participants felt learning how to use the tool was facilitated by its similarity to other tools, and to other workflow processes in Epic. One barrier that a few participants raised was that if the CDS alert for the EHR CT CDS tool did not trigger automatically, they were unsure of how to manually activate it.

Maintenance

Maintenance is the extent to which a program or policy becomes part of routine organizational practice and culture.[13] The interviews noted that the EHR CT CDS tool fit well within the institutional culture and thus became so integrated into the clinical workflow at each of the PECARN sites that the tool was still in active use despite the completion of the trial. The interviewees reported actual and potential reinforcing factors to motivate health care providers to maintain their use of the CDS, including a belief that the EHR CT CDS tool was educational both for health care providers (specifically residents, nurse practitioners, and physician assistants) and families. The belief was that the use of the EHR CT CDS tool engendered greater confidence in health care provider communication with family members. There was also a strong sense that the EHR CT CDS tool was useful for supporting education with families. The tool also helped some health care providers feel more comfortable safely discharging children home by providing the health care providers with reassurance about their clinical judgment.

A proposed reinforcing factor ([Table 3] and [Fig. 1]) that may increase use of the EHR CT CDS tool was a benchmark report which would compare use of the tool between providers, potentially generating extrinsic motivation to use it as intended. A frequently proposed reinforcing factor was the suggestion to develop family-friendly visualizations and output to support clear end-user communication with family members. Automating the output of the EHR CT CDS tool into discharge instructions was another suggestion for reinforcing use of the tool. For example, when the EHR CT CDS tool output indicated very low risk of TBI, it should be incorporated into the discharge instructions; when it indicated risk other than very-low risk, that information could activate an observation pathway.

Proposed barriers to the maintenance of the EHR CT CDS tool included the technical challenges of migrating the output of the tool into the clinical note. Health care providers believed that clearer training on how to automatically include the EHR CT CDS tool output into the note might sustain the use of the tool. Another barrier to maintenance was the lack of clarity as to which health care provider should complete the EHR CT CDS tool. In the current implementation of the EHR CT CDS tool, there was variation across sites regarding which providers completed the EHR CT CDS tool to allow for flexible integration into each site's workflow.

Discussion

Using the RE-AIM framework, this study evaluated factors that support the implementation and dissemination of the EHR CT CDS tool into emergency medical care to scale it to other EDs in the future. The results provide concrete strategies for the adoption, implementation, and maintenance of the EHR CT CDS tool to improve its application into practice. [Fig. 1] summarizes the methods for the study (including data sources, settings, and sites), themes from the qualitative analysis, and future suggestions for the implementation of the EHR CT CDS tool. These data address one of the largest challenges with CDS: namely, disseminating best practices for the design, development, implementation, maintenance, and evaluation of these tools so that health care institutions can learn from one another.[20]

Adoption

Based on our data, successful adoption of the EHR CT CDS tool was strongly supported by close alignment with the institutional culture of the organizations participating in the PECARN and CREST network, which value evidence-based practice and are open to implementing informatics-based CDS tools. According to a recent meta-synthesis of qualitative studies, one of the most common issues with adoption of CDS is poor clinician–patient–tool integration of the CDS within the sociotechnical system.[22] This barrier was addressed prior to adoption of the EHR CT CDS tool through extensive analysis of the sociotechnical environment at the sites.[10]

Implementation

Usability is among the most common barriers to the implementation of CDS tools, including issues related to the nuisance of alerts and system immaturity (i.e., interoperability).[22] These barriers were considered and addressed prior to the adoption and implementation phases through the systematic development of the prediction rules and EHR CT CDS tool.[5] [6] [9] [10] [23] [24] The implementation of the EHR CT CDS tool was supported by a strong perceived relevance among stakeholders, and an active clinical champion to support its use and value. Our finding of the importance of an active clinical champion for the successful CDS implementation is consistent with findings from a prior study conducted in the CREST Network that evaluated a CDS tool for managing pulmonary embolisms.[26] During the implementation process of the EHR CT CDS tool, important features that were identified as critical to the success of CDS implementation were carefully attended to, including: providing the tool to health care providers based on the primary chief complaint (rather than having them seek it out); having it integrated with the charting and order entry system (rather than a stand-alone system); and providing a specific recommendation and the decision support at the time and location of decision making.[26]

Maintenance

In general, when moving an informatics application from a clinical trial into routine practice, seamless integration into clinician workflow is important so that it is not perceived as an additional disruptive task. In this case, the EHR CT CDS tool was so well integrated into the clinical workflow that there was little interest in turning it off after the study was completed. Moreover, clinician perception that the tool was useful reinforced its use. In another RE-AIM analysis completed for a multiple sclerosis falls-prevention intervention, factors that influenced long-term maintenance were: attentiveness to branding and promotion of the intervention; using behavior change theory to guide intervention development and delivery; building ongoing support, and small reminders; and documenting the cost-effectiveness and cost-benefit of the intervention.[27] Though the EHR CT CDS tool implementation did not include all of these aspects to support long-term maintenance of tool use, its development was guided by behavior change theory, and it had ongoing support and reminders from a clinical champion.

To support the maintenance of the tool, the suggestion from health care providers to receive feedback of their utilization of the CDS tool (and CT rates) compared with their peers (benchmark report) is consistent with best practices for performance improvement and driving knowledge translation.[28] [29] [30] The optimal feedback loop appears to be repeated feedback from multiple modalities by both supervisors and peers.[29] [30]

Limitations

One of the limitations to understanding the reach of the multicenter PECARN implementation clinical trial was the challenge comparing the clinical and demographic characteristics of the patients in the trial with robust national statistics. At this time, national statistics are not up-to-date and the characterization of the head injuries and TBIs are not as granular as in the PECARN data. Limitations of the efficacy of the study are published elsewhere.[6] There were some differences in the implementation of the study at each site, but these differences were part of the intended variation within a pragmatic clinical trial design. For the qualitative interviews, there may have been some selection bias related to interviewing a sample of health care providers who were engaged in the PECARN and from within the CREST Network who opted to be interviewed. In addition, our study was focused on results across sites rather than between groups of sites (i.e., academic vs. community EDs). More broadly, another limitation to generalizability is all of the study sites belonged to a research network, so they may have been more inclined to have evidence-based practices and incorporate more innovation into their practice. On the other hand, the strength of the study is that both academic medical centers and community hospitals provided data and enrolled patients. Another strength of the study is that the CDS was developed and tested in Epic, an EHR with substantial penetration (reach), thus providing support for scaling the intervention. In addition, provider-specific selection bias was mitigated by automatically triggering the CDS alert based on chief complaints.

Future Implications

By providing the methods for implementation, we see many logical next steps for this EHR CT CDS tool, as well as others like it. Throughout the interviews, health care providers voiced interest in applying the same methodology for diagnoses such as abdominal pain, asthma, otitis media, fever, and appendicitis. Another suggestion for the current tool is to provide tailored messaging to different types of health care members. Overall, one physician summarized it well with a common sentiment on perceived impact; “I think getting it out there to beyond the pediatric emergency department is probably where the big impact will be felt.” This was followed up by specific recommendations to get the EHR CT CDS tool into urgent care clinics because many referrals come from urgent care for which a CT scan is not indicated. Next steps for this research also include evaluating whether CDS tools reduce unnecessary health care expenditure and improve the efficiency of patient flow through the ED by decreasing the duration of the evaluation and shortening time to discharge. More research is also needed on differences between academic and community EDs. Finally, further study could clarify if this EHR CT CDS tool helps to decrease anxiety for children and parents.

Conclusion

We identified multiple concrete features to facilitate and support the adoption, implementation, and maintenance of the EHR CT CDS tool for children with minor blunt head trauma.[6] These features can support the future implementation of the EHR CT CDS tool into general and pediatric EDs around the country.

Multiple Choice Questions

1. What are the benefits of applying an implementation framework to complement the findings of an efficacy study?

   • Provide additional information about factors that influenced adoption, implementation, and maintenance.

   • Generate best practices for future implementation studies.

   • Advance the science of implementation and dissemination.

   • All of the above.

   Correct Answer: The correct answer is option d.

2. What is an example of a predisposing factor for the implementation of the CT CDS tool in emergency departments?

   • Institutional investment in user training.

   • Approachable clinical champion.

   • Integration into the clinical workflow.

   • Ease of use of the CT CDS tool.

   Correct Answer: The correct answer is option b, Approachable clinical champion. Responses a, c, and d, are three enabling, not predisposing, factors for implementation. Predisposing factors occur before the behavior and influence motivation to undertake a particular behavior.

Conflict of Interest

None.

Acknowledgments

We greatly appreciate the organization of the present study by the research coordinators at participating sites. We would like to thank Dr. Sunmoo Yoon for developing the infographic displayed in Figure 1.

Protection of Human and Animal Subjects

Written or verbal informed consent was obtained at the beginning of each interview depending on the local Institutional Review Board requirements at each site.

• References

• 1 Centers for Disease Control and Prevention. Rates of TBI-related emergency department visits, hospitalizations, and deaths — United States, 2001–2010. Secondary Rates of TBI-related emergency department visits, hospitalizations, and deaths — United States, 2001–2010; 2016 . Available at: https://www.cdc.gov/traumaticbraininjury/data/rates.html . Accessed August 13, 2018
• 2 Center for Disease Control and Prevention. Rates of TBI-related emergency department visits by age group — United States, 2001–2010. Secondary Rates of TBI-related emergency department visits by age group — United States, 2001–2010; 2016 . Available at: https://www.cdc.gov/traumaticbraininjury/data/rates_ed_byage.html . Accessed August 13, 2018
• 3 Miglioretti DL, Johnson E, Williams A. , et al. The use of computed tomography in pediatrics and the associated radiation exposure and estimated cancer risk. JAMA Pediatr 2013; 167 (08) 700-707
• 4 Marin JR, Weaver MD, Barnato AE, Yabes JG, Yealy DM, Roberts MS. Variation in emergency department head computed tomography use for pediatric head trauma. Acad Emerg Med 2014; 21 (09) 987-995
• 5 Kuppermann N, Holmes JF, Dayan PS. , et al; Pediatric Emergency Care Applied Research Network (PECARN). Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study. Lancet 2009; 374 (9696): 1160-1170
• 6 Dayan PS, Ballard DW, Tham E. , et al; Pediatric Emergency Care Applied Research Network (PECARN); Clinical Research on Emergency Services and Treatment (CREST) Network; and Partners Healthcare; Traumatic Brain Injury-Knowledge Translation Study Group. Use of traumatic brain injury prediction rules with clinical decision support. Pediatrics 2017; 139 (04) e20162709
• 7 Tham E, Swietlik M, Deakyne S. , et al; Pediatric Emergency Care Applied Research Network (PECARN). Clinical decision support for a multicenter trial of pediatric head trauma. Appl Clin Inform 2016; 7 (02) 534-542
• 8 Goldberg HS, Paterno MD, Grundmeier RW. , et al. Use of a remote clinical decision support service for a multicenter trial to implement prediction rules for children with minor blunt head trauma. Int J Med Inform 2016; 87: 101-110
• 9 Deakyne SJ, Bajaj L, Hoffman J. , et al; Pediatric Emergency Care Applied Research Network (PECARN). Development, evaluation and implementation of chief complaint groupings to activate data collection: a multi-center study of clinical decision support for children with head trauma. Appl Clin Inform 2015; 6 (03) 521-535
• 10 Sheehan B, Nigrovic LE, Dayan PS. , et al; Pediatric Emergency Care Applied Research Network (PECARN). Informing the design of clinical decision support services for evaluation of children with minor blunt head trauma in the emergency department: a sociotechnical analysis. J Biomed Inform 2013; 46 (05) 905-913
• 11 Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999; 89 (09) 1322-1327
• 12 Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res 2006; 21 (05) 688-694
• 13 Bakken S, Ruland CM. Translating clinical informatics interventions into routine clinical care: how can the RE-AIM framework help?. J Am Med Inform Assoc 2009; 16 (06) 889-897
• 14 Green LW, Kreuter MW. Health Promotion Planning: An Educational and Ecological Approach. 4th ed. Boston, MA: McGraw-Hill; 2005
• 15 Holmes-Rovner M, Valade D, Orlowski C, Draus C, Nabozny-Valerio B, Keiser S. Implementing shared decision-making in routine practice: barriers and opportunities. Health Expect 2000; 3 (03) 182-191
• 16 Aronsky D, Chan KJ, Haug PJ. Evaluation of a computerized diagnostic decision support system for patients with pneumonia: study design considerations. J Am Med Inform Assoc 2001; 8 (05) 473-485
• 17 Faul MXL, Wald MM, Coronado VG. . Traumatic Brain Injury in the United States: Emergency Department Visits, Hospitalizations and Deaths 2002–2006. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Injury Prevention and Control; 2010
• 18 EMR 2016: The Market for Electronic Medical Records by Kalorama Information. Available at: https://www.kaloramainformation.com/EMR-Electronic-Medical-10009693/ . Accessed August 27, 2018
• 19 Monegain B. . Kalorama: Cerner, McKesson earn EHR marketshare lead, Epic and Allscripts follow. Secondary Kalorama: Cerner, McKesson earn EHR marketshare lead, Epic and Allscripts follow; 2016 . Available at: http://www.healthcareitnews.com/news/kalorama-cerner-mckesson-earn-ehr-marketshare-lead-epic-and-allscripts-follow . Accessed August 13, 2018
• 20 Epic software. Epic fact sheet. In: Communications ET, ed.; 2017
• 21 Sittig DF, Wright A, Osheroff JA. , et al. Grand challenges in clinical decision support. J Biomed Inform 2008; 41 (02) 387-392
• 22 Miller A, Moon B, Anders S, Walden R, Brown S, Montella D. Integrating computerized clinical decision support systems into clinical work: a meta-synthesis of qualitative research. Int J Med Inform 2015; 84 (12) 1009-1018
• 23 Hess EP, Wyatt KD, Kharbanda AB. , et al. Effectiveness of the head CT choice decision aid in parents of children with minor head trauma: study protocol for a multicenter randomized trial. Trials 2014; 15: 253
• 24 Bressan S, Romanato S, Mion T, Zanconato S, Da Dalt L. Implementation of adapted PECARN decision rule for children with minor head injury in the pediatric emergency department. Acad Emerg Med 2012; 19 (07) 801-807
• 25 Ballard DW, Vemula R, Chettipally UK. , et al; KP CREST Network Investigators. Optimizing clinical decision support in the electronic health record. clinical characteristics associated with the use of a decision tool for disposition of ED patients with pulmonary embolism. Appl Clin Inform 2016; 7 (03) 883-898
• 26 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
• 27 Finlayson M, Cattaneo D, Cameron M. , et al. Applying the RE-AIM framework to inform the development of a multiple sclerosis falls-prevention intervention. Int J MS Care 2014; 16 (04) 192-197
• 28 Meeker D, Linder JA, Fox CR. , et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial. JAMA 2016; 315 (06) 562-570
• 29 Ivers N, Jamtvedt G, Flottorp S. , et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012; (06) CD000259
• 30 Le Grand Rogers R, Narvaez Y, Venkatesh AK. , et al. Improving emergency physician performance using audit and feedback: a systematic review. Am J Emerg Med 2015; 33 (10) 1505-1514