Development and Evaluation of a Clinical Decision Support System to Improve Medication Safety

Abstract

Background Clinical decision support systems (CDSSs) are a good strategy for preventing medication errors and reducing the incidence and severity of adverse drug events (ADEs). However, these systems are not very effective and are subject to multiple limitations that prevent their implementation in clinical practice.

Objectives The objective of this study was to evaluate the effectiveness of an advanced CDSS, HIGEA, which generates alerts based on predefined clinical rules to identify patients at risk of an ADE.

Methods A multidisciplinary team defined the system and the clinical rules focusing on medication errors commonly encountered in clinical practice. Four intervention programs were defined: (1) dose adjustment in renal impairment; (2) adjustment of anticoagulation/antiplatelet therapy; (3) detection of biochemical/hematologic toxicities; and (4) therapeutic drug monitoring. We performed a 6-month observational prospective study to analyze the effectiveness of these clinical rules by calculating the positive predictive value (PPV).

Results The team defined 211 clinical rules. During the study period, HIGEA generated 1,086 alerts (8.9 alerts per working day), which were reviewed by pharmacists. Fifty-one percent (554/1,086) of alerts generated an intervention to prevent a possible ADE; of these, 66% (368/554) required a documented modification to therapy owing to a real prescription error intercepted. The intervention program that induced the highest number of modifications to therapy was the dose adjustment in renal impairment program (PPV = 0.51), followed by the adjustment of anticoagulation/antiplatelet therapy program (PPV = 0.24). The percentage of accepted interventions was similar in surgical units (68%), medical units (67%), and critical care units (63%).

Conclusion Our study offers evidence that HIGEA is highly effective in preventing potential ADEs at the prescription stage.

Keywords

clinical decision support - adverse drug event - safety - pharmacy - alerting

Background and Significance

Medication errors significantly increase patient morbidity and mortality and cause considerable increases in costs in health care institutions.[1] Most are considered to be preventable.[2] [3]

Medication errors can occur at any point in the medication use process. However, research has shown that they are more common at the prescription stage (39%).[4] Different approaches have been adopted to help physicians minimize these errors, including educational strategies and use of advanced health information technologies. Major emphasis has been placed on implementing computerized physician order entry (CPOE) combined with a clinical decision support system (CDSS).[5] [6] [7] [8] [9] CDSS provides clinicians with clinical knowledge and patient-specific information in order to optimize the safety and quality of the pharmacological treatment prescribed. These systems are associated with reduced morbidity rates, improved prescribing practices, improved patient monitoring, reduced health care costs, and reduced adverse drug event (ADE) rates.[10] [11] [12] [13]

Optimal use of CPOE with CDSS requires integration of multiple clinical information systems, including medical records, clinical laboratory software, and pharmacy-based software. This is not a concern for hospitals where the electronic health record already integrates clinical data. However, in hospitals such as ours, where several clinical information systems are being used, data integration has proven to be challenging.

During the development and implementation of the CDSSs, other numerous barriers are encountered. The majority of CDSSs are not able to integrate the information systems in real time or to translate clinical guidelines into appropriate alerts.[14] [15] The lack of sensitivity and specificity of the alerts, together with the inability to customize them, often results in a high rate of override.[16]

In 2015, our institution developed a CDSS (HIGEA) that benefits from the integration of multiple hospital information systems in real time and generates patient-specific alerts to prevent potential ADEs based on predefined consensual clinical rules (CRs). Alerts are reviewed by the pharmacists during their ward-based activities in order to prevent alert fatigue during prescription. CRs can be redesigned and customized in order to improve their effectiveness.

Objectives

The objective of the present study was to evaluate the effectiveness of this automated/integrated real-time CDSS in the prevention of potential ADEs.

Methods

Setting

Our hospital is a 1,300-bed tertiary teaching institution employing more than 8,000 professionals who are responsible for the direct specialist health care of a catchment population of approximately 350,000 people. [Table 1] shows the most significant indicators of hospital care activity between January 2016 and June 2016.

Table 1 Indicators of hospital care activity between January 2016 and June 2016

Indicator	N
Hospitalizations	25,449
Hospital stays	190,799
Average length of stay	7.5 d
Surgical procedures	16,772
Medical consultations	329,487
Drug prescriptions/day	11,500

In our hospital, physicians enter all the prescriptions into an in-house CPOE system, which was traditionally supported by a basic CDSS (Farhos Prescriplan – Visual Limes, Spain). This safety monitoring tool is based on protocols, standardized doses for most drugs, and alerts in case of allergies, duplications, and interactions. The process is automated using a real-time alert system. When a medication is prescribed, an alert generated in the system warns the doctor of any possible medication error. Immediately after prescription, the CPOE system is checked for new or modified medication orders placed by the pharmacist. The pharmacist ensures continuous centralized order verification. Twelve pharmacists are responsible for patient care areas during the day shift; each pharmacist covers one specialty area. Dispensing is facilitated by profiled automated dispensing cabinets (Pyxis, Grifols), from which nurses can withdraw medication once it has been prescribed and verified by the clinical pharmacist.

Design of the Advanced CDSS: HIGEA

HIGEA was designed by a multidisciplinary team comprising 10 permanent members. The pharmacy department was represented by four pharmacists from different areas of expertise (medical, surgical, critical care, and a dedicated ADE pharmacist) and the medical staff by a nephrologist, a hepatologist, a hematologist, and an infectious disease specialist. The team was completed by two information systems specialists. Team members were selected based on their extensive clinical experience. In any case, all members were asked to present their colleagues' opinions and to consult them if necessary during the development of the system.

HIGEA integrates multiple hospital information systems in real time and generates patient-specific alerts to prevent ADEs based on predefined CRs ([Fig. 1]).

Fig. 1 Design of clinical decision support system.

The integration was designed to receive Health Level Seven (HL7) messages from laboratory systems and the CPOE. A Mirth server was setup, and several channels were set to work in real time in order to receive the information sent from these systems. A few seconds after a message is received, a Web service is available to provide information regarding existing alerts for the patient in the message. Therefore, if a physician prescribes a new drug or new laboratory information is generated, HIGEA automatically activates or deactivates the alert. An average volume of 50,000 messages are processed daily.

The information was transformed to a “standard” data structure in a NoSQL database, where all information of a given patient is stored as a document. This data store format enables us to check very quickly whether a given patient satisfies a specific rule or not, as complete patient information is retrieved at once. Unlike other, similar solutions, which are based on traditional SQL systems, HIGEA does not need to perform SQL joins to find information on a given patient, thus saving processing time and improving real-time responses.

The system was developed with the following features:

1. Integration of clinical laboratory data and the CPOE system. Laboratory values (biochemistry, hematology, immunology, and genetics) were imported from the laboratory online results program, Modulab. Data on patient characteristics and drug use were imported from the CPOE, Farhos Prescriplan. The integration was performed using a standard language (HL7) and algorithms that create a homogeneous knowledge base to process CRs.

2. Generation of alerts by combining data from the electronic patient databases (laboratory data and CPOE) and the CR bundle previously defined by the multidisciplinary team. The detection system ran in real time and performed searches for new orders or new laboratory test values.

3. Generation of standardized recommendations for the pharmacists' interventions. The multidisciplinary team agreed on the advice that the pharmacist had to give to prescribers.

4. Automatic recording of the pharmacists' interventions and their acceptance by the physicians. The interventions that pharmacists implemented to prevent a potential ADE in response to the alerts generated by HIGEA were automatically recorded by the Web service in real time. This collects and stores all the information received by the information systems. Thus, the pharmacist only reviews the information recorded by the system.

5. Generation of dashboards that allow analysis of data and a systematic evaluation of the usefulness of the system.

6. Automatic prioritization of alerts based on their impact. In order to prevent alert fatigue, the rules were continually assessed by evaluating its positive predictive value (PPV). Rules that have more impact on patients are highlighted and shown first in the Web service. The pharmacist can then review the most relevant alerts first.

Definition of Clinical Rules: A Consensus-Based Process

A CR consists of the observation of ≥ 1 clinical positive and/or negative condition (e.g., a drug prescription and the presence of a laboratory value) and a standardized recommendation to change the treatment in order to prevent an ADE. A CR bundle was defined for four different intervention programs: (1) program for dose adjustment in renal impairment; (2) program for adjustment of anticoagulation/antiplatelet therapy; (3) program for detection of biochemical/hematologic toxicities; and (4) therapeutic drug monitoring program (detecting inappropriate blood levels of high-risk drugs).

A dedicated ADE pharmacist performed a bibliographic search to identify possible CRs for each of the four intervention programs defined. These CRs were presented to the multidisciplinary team in order to identify those that, according to clinical practice, could be more effective in the prevention of ADEs. The team prioritized those CRs that were considered more harmful, less known by physicians, less prevalent, or more complex. The final definition of the bundles required 8 meetings, 2 for each program, with each lasting approximately 3 hours. In addition, a 2-hour introductory session was held to explain how the CDSS operated.

Software Validation

After building the CRs into HIGEA, a validation process was carried out over 3 months to establish the technical correctness of the system. The pharmacists reviewed the alerts reported by HIGEA and, after checking the patient's clinical information, ensured that the system adequately detected patients described in the expected situations in the CR. They then contacted the prescribing physician to recommend changes in treatment.

This process ensured that the system does not generate false-positive or false-negative alerts. Once it was validated that the software fully satisfied all expected requirements, it was implemented in daily clinical practice.

Validation Study

In order to assess the true impact of alerts generated by the CRs, we performed an observational nonrandomized prospective study from January 1, 2016 to June 30, 2016.

The PPV was analyzed as an indicator of the effectiveness of the previously defined CRs. The PPV is the probability that an alert can prevent a potential ADE, namely as the ratio of modifications in treatment to alerts reviewed. It measures the real impact of a CR on a patient, which is theoretical.

PPV = alerts with an accepted intervention / total of alerts.

The total number of alerts generated by the system, the prescribing errors intercepted, and the response to the intervention were analyzed. The specific intervention program that enabled the detection of the potential ADE and the patient's location by department was also identified.

This study was conducted in accordance with the Declaration of Helsinki and was approved by the ethics committee of the Gregorio Marañón Hospital before the study began.

Results

The team defined 211 CRs: 110 for the “program for dose adjustment in renal impairment,” 24 for the “program for adjustment of anticoagulation/antiplatelet therapy,” 64 for the “program for detection of biochemical/hematologic toxicities,” and 13 for the “therapeutic drug monitoring program.” Based on these CRs, HIGEA generated a total of 1,086 alerts (8.9 alerts per working day) during the study period. The impact of the alerts in terms of the number of interventions generated and the grade of adherence to the recommendations are shown in [Fig. 2].

Fig. 2 Number of alerts and rate of intervention.

The pharmacist reviewed all the alerts and approximately half of the alerts (554 out of 1,086 [51%]) generated an intervention to prevent a possible ADE. Of these, 368 interventions were accepted by health care professionals and led to a change in treatment (e.g., changes in dose or frequency or route of administration, end of treatment). Overall, 368 of 1,086 alerts (34%) required a documented modification to therapy because of a real prescription error intercepted. No advice was given in 532 of 1,086 alerts (49%). In these cases, the pharmacist evaluated the patient's clinical status by consulting the clinical history and contacting the prescribing physician. Then, the prescribed treatment was considered adequate (e.g., furosemide and sodium [Na] < 125 mmol/L: furosemide can induce hyponatremia but is indicated in patients with hypoosmolar hyponatremia; levofloxacin and estimated glomerular filtration rate [eGFR] < 50 mL/min: a patient with eGFR = 30 mL/min and levofloxacin 250 mg/12 h is correct for a patient with complicated pneumonia but not for a patient with bronchitis).

[Table 2] shows all the results for alerts generated by intervention program and the global PPV in each intervention program. The CRs with the highest PPV in each intervention program are detailed in [Supplementary Table S1] (available in the online version).

Table 2 Positive predictive value of alerts generated by intervention program

Outcome	Alerts (N)	Alerts with intervention (N)	Treatment changes induced (N)	Positive predictive value
Intervention program
Program for adjustment of dose in renal impairment	430	306	218	0.51
Program for adjustment of anticoagulation/antiplatelet therapy	395	145	94	0.24
Program for detection of biochemical/hematologic toxicities	212	93	46	0.22
Therapeutic drug monitoring program	49	10	10	0.20
Total	1,086	554	368

[Fig. 3] shows the analysis of alerts generated in each program stratified by type of unit.

Fig. 3 Acceptance rate in each intervention program stratified by type of unit.

Overall, the highest number of alerts with an intervention (396; 71%) was generated for patients admitted to medical units, followed by surgical units (88; 16%) and critical care units (70; 13%). The percentage of accepted interventions was similar in surgical units (68%), medical units (67%), and critical care units (63%).

The most frequent individual alerts that led to changes in treatment were generated in response to enoxaparin subcutaneous and eGFR < 30 mL/min (55/368 changes), ranitidine orally and eGFR < 50 mL/min (39/368 changes), meropenem and eGFR < 50 mL/min (26/368 changes), omeprazole and Na < 125 mmol/L (20/368 changes), and levofloxacin intravenous and eGFR < 50 mL/min (18/368 changes).

Discussion

We describe the implementation of an advanced CDSS and evaluate its utility in identifying potential ADEs. Specifically, we confirm that real-time integration of clinical information provides a highly efficient CDSS for improving medication safety. When specialist pharmacists examine the alerts generated daily, the system identifies a large number of potential ADEs that go unnoticed by the physician and led to a change in the clinician's decision (368/554; 66%). These highly effective alerts are prepared to be incorporated into the CPOE.

The major benefit of this solution is that it combines data from various clinical information systems (clinical laboratory data and CPOE system), whereas most other CDSS are severely limited by nonintegrated information systems[17] [18] [19] [20] and, therefore, less specific rules that are restricted to a single criterion such as prescription of a medication or an abnormal laboratory result. These rules have high false-positive rates because their occurrence is rarely related to an ADE. The PPV of these rules is < 0.1, which is much lower than our values (0.34). Recently published studies on medication-related CDSS identify the increase in the sensitivity and specificity of alerts as an area that can be improved. This improvement should include more patient-specific information in order to reduce alert fatigue.[21] [22]

Second, alerts run in real time, and the system continuously performs searches for new orders and laboratory test values. This feature differs from many commercially available CDSS, in which real-time analysis is not available. Continuous searching also helps to obtain more efficient CRs with a better PPV, because the alerts always show the real clinical situation of the patient (the most recent drug prescription and laboratory test value).

It is also noteworthy that HIGEA uses CRs to detect patients at risk of an ADE instead of using alerts to identify ADEs that have already occurred. This approach differs from other software applications in which rules commonly include a toxic serum drug level or prescription of an antidote.[23] [24] [25] [26] [27] [28] [29] [30] In fact, Silverman et al[30] demonstrated that after modification of their CRs from detecting actual ADEs to identifying potential ADEs, the volume of interventions by the pharmacists increased and the rules became more efficient.

HIGEA was designed as a customizable system, which allows the user to modify the CRs in order to increase their efficiency. Other similar systems described by the scientific literature do not include this facility, and the incorporated commercial bundle of CRs cannot be modified by the user.[30] [31]

Finally, another major benefit of HIGEA is that it was provided with a large set of CRs that covers a wide range of clinical conditions and provides a specific dosage recommendation for the new prescription.[32] Only 10% of published studies in this area[15] use rules that have been defined according to clinical needs. In our case, the CRs were defined by a multidisciplinary team that applied a structured method for the various intervention programs. It has already been demonstrated that an updated and consistent knowledge base positively affects the efficiency of these systems.[31] The “program of dose adjustment in renal impairment” was prioritized because of the high number of drugs that have to be adjusted in patients with renal impairment and the very scant knowledge of these drugs among health professionals. In fact, it has been demonstrated that the rate of appropriate drug prescribing in kidney impairment is low and that the use of a CDSS could improve patient safety.[32] [33] The “program for adjustment of anticoagulation/antiplatelet therapy” was prioritized because of the severe events that an error with such high-risk drugs could lead to. The “program for detection of biochemical/hematologic toxicities” was included because most of these ADEs are not often taken into account owing to their low prevalence. The “therapeutic drug monitoring program” was included because the alert system can shorten the time to response once the drug level is available.

Although the percentage of safety alerts resulting in an intervention by the hospital pharmacist (51%) may seem low, it is significantly higher than the percentages observed with other CDSS.[15] [34] [35] Nanji et al[36] evaluated the rate of alert overrides (alerts without an intervention) and its appropriateness. Surprisingly, and consistent with our results, they found that about half of the CDSS alerts were overridden by providers and consequently did not result in an intervention. The highest number of alerts with an intervention was generated for patients admitted to medical units (71%). Given the considerable effort necessary to implement a new technology and in view of these results, this type of patient would be prioritized.

Two-thirds (66%) of the interventions were accepted by physicians; this finding is consistent with the percentages observed in other studies. In our case, we did not identify significant differences between medical, surgical, and critical care units (63–68%). Jha et al[34] investigated a commercial computerized surveillance system (Vigilanz Corporation – Dynamic Pharmacovigilance) over a 4-month period and found that the physician was contacted in 30 of 266 reviewed alerts (11.3%). The acceptance rate of these 30 interventions was 50%. Kilbridge and Ahmad[35] evaluated 4,604 triggers from a computer-based ADE surveillance system over a 2-month period; of these, 260 led to an intervention (4%). After analyzing the Theradoc computer-based monitoring system, Silverman et al[30] reported an intervention rate of 5 to 13% and an acceptance rate of 78 to 92%. Rommers et al[37] evaluated the use of a CDSS (“ADEAS”) similar to ours over a period of 5 months. In this case, the system generated 2,650 alerts, of which 204 led to an intervention (7.7%). The percentage of acceptance was 63%.

It should be noted that the acceptance percentages are low, probably owing to the complexity of the clinical situation of some patients, which sometimes forces us not to follow standard recommendations in daily clinical practice.

Finally, it is noteworthy that none of these studies were performed in Europe.[15] Despite growing evidence of the positive clinical impact of health technologies on safety, their adoption and implementation are very slow in some countries. In Spain, according to a survey conducted by the Spanish Society of Hospital Pharmacy in 2015, only 20% of hospitals had implemented a CDSS aimed at increasing the safety of pharmacotherapy.[38] High cost is a major barrier, since hospitals must make a large initial investment with no clear return. For these reasons, there is an urgent need to test CDSS in Europe because of significant structural differences in health systems between both regions. We consider that our study provides relevant results for making strategic decisions concerning implementation of measures to increase patient safety.

HIGEA has been designed in such a way that it can be exported to other institutions; both the software and the CRs were validated in this study. In fact, three Spanish hospitals are already using it.

Limitations

First, not all potential ADEs can be detected with this technology, as some are related to clinical information expressed using natural language, as is the case with diagnosis and symptoms. This information is not easily accessible, although it has already proven to be very useful for other authors.[39] [40] Since HIGEA does not fully integrate the electronic health record, the detection of such ADEs is not possible and, consequently, some CRs present a low PPV. We are currently working to improve reasoning by including natural language processing and identification of semantic entities from unstructured information in the electronic health record.

Second, HIGEA was developed as a CDSS within the CPOE. However, to evaluate its effectiveness in the prevention of ADE it is necessary guarantee the analysis of all the generated alerts, which will allow us to reliably calculate the PPV for each of the CRs. For this reason, the alerts generated by the system are reviewed by the pharmacists and not directly by the prescribing physician. We believe that prescreening of alerts by a clinical pharmacist reduce alert fatigue during prescription, and could increase the likelihood of appropriate prescribing of these medications.[10] [31] [41] [42] In fact, a review from van der Sijs et al[16] showed that physicians ignore safety alerts in 49 to 96% of cases. In the future, some alerts might be better presented as online alerts for the physician when immediate action is necessary, that is, those with PPV = 1.

Finally, given that the recommendations defined in the program for adjustment of dose in renal impairment do not include specific indications for obese or hemodialysis patients, these groups were excluded from the study.

Conclusion

HIGEA, an integrated real-time CDSS, is highly effective in preventing potential ADEs. The clinical pharmacist has played a key role in the success of the system. Our study offers evidence that customization of CRs significantly improves the safety and quality of health care decisions when a multidisciplinary team is involved.

Clinical Relevance Statement

HIGEA is a CDSS that makes it easier for health care professionals to identify patients with a high risk of experiencing an ADE, thanks to real-time integration of various hospital clinical information systems.

Multiple Choice Questions

1. Which of the following CDSS is more efficient in the detection of potential ADEs?

   • A CDSS that receives the drug prescription information.

   • A CDSS that receives the drug prescription information and clinical laboratory data.

   • A CDSS that receives the drug prescription information, clinical laboratory data, and electronic health record information.

   • A CDSS that receives clinical laboratory data.

   Correct Answer: The correct answer is option c, because the system combines data from many hospital information systems, thus enabling the use of clinical rules with a higher positive predictive value.

2. What is one of the major benefits of an advanced clinical decision support system?

   • It provides clinicians with clinical knowledge.

   • It enables drug prescription by physicians.

   • It generates alerts by combining data from the electronic patient database.

   • a and c are correct.

   Correct Answer: The correct answer is option d, because these are the two functionalities that differentiate our system from a simple clinical decision support system.

Conflict of Interest

A.H.-A reports grants and other from Instituto de Salud Carlos III, Ministerio de Ciencia, Innovación y Universidades, during the conduct of the study. B.M.-A reports grants and other from Instituto de Salud Carlos III, Ministerio de Ciencia, Innovación y Universidades, during the conduct of the study. C.R.-G reports grants and other from Instituto de Salud Carlos III, Ministerio de Ciencia, Innovación y Universidades, during the conduct of the study. M.L.M.-B reports other from Instituto de Salud Carlos III, Ministerio de Ciencia, Innovación y Universidades, during the conduct of the study. M.S.S. reports grants and other from Instituto de Salud Carlos III, Ministerio de Ciencia, Innovación y Universidades, during the conduct of the study. S.I.-G reports grants and other from Instituto de Salud Carlos III, Ministerio de Ciencia, Innovación y Universidades, during the conduct of the study. V.E.-V. reports grants and other from Instituto de Salud Carlos III, Ministerio de Ciencia, Innovación y Universidades, during the conduct of the study. All other authors declare that they have no conflicts of interest in the research.

Protection of Human and Animal Subjects

This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by Gregorio Marañón Hospital Institutional Review Board.

• References

• 1 Kohn LTCJ, Donaldson MS. To Err is Human: Building a Safer Health System. Washington, DC: National Academy Press; 1999
• 2 Leape LL, Brennan TA, Laird N. , et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991; 324 (06) 377-384
• 3 Bates DW, Cullen DJ, Laird N. , et al; ADE Prevention Study Group. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. JAMA 1995; 274 (01) 29-34
• 4 Leape LL, Bates DW, Cullen DJ. , et al; ADE Prevention Study Group. Systems analysis of adverse drug events. JAMA 1995; 274 (01) 35-43
• 5 Bobb A, Gleason K, Husch M, Feinglass J, Yarnold PR, Noskin GA. The epidemiology of prescribing errors: the potential impact of computerized prescriber order entry. Arch Intern Med 2004; 164 (07) 785-792
• 6 Lewis PJ, Dornan T, Taylor D, Tully MP, Wass V, Ashcroft DM. Prevalence, incidence and nature of prescribing errors in hospital inpatients: a systematic review. Drug Saf 2009; 32 (05) 379-389
• 7 Franklin BD, Reynolds M, Shebl NA, Burnett S, Jacklin A. Prescribing errors in hospital inpatients: a three-centre study of their prevalence, types and causes. Postgrad Med J 2011; 87 (1033): 739-745
• 8 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 (12) 1409-1416
• 9 Nuckols TK, Smith-Spangler C, Morton SC. , et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst Rev 2014; 3: 56
• 10 Kuperman GJ, Bobb A, Payne TH. , et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007; 14 (01) 29-40
• 11 Bright TJ, Wong A, Dhurjati R. , et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
• 12 Kaushal R, Jha AK, Franz C. , et al; Brigham and Women's Hospital CPOE Working Group. Return on investment for a computerized physician order entry system. J Am Med Inform Assoc 2006; 13 (03) 261-266
• 13 Wolfstadt JI, Gurwitz JH, Field TS. , et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med 2008; 23 (04) 451-458
• 14 Rommers MK, Zegers MH, De Clercq PA. , et al. Development of a computerised alert system, ADEAS, to identify patients at risk for an adverse drug event. Qual Saf Health Care 2010; 19 (06) e35
• 15 Forster AJ, Jennings A, Chow C, Leeder C, van Walraven C. A systematic review to evaluate the accuracy of electronic adverse drug event detection. J Am Med Inform Assoc 2012; 19 (01) 31-38
• 16 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
• 17 Azaz-Livshits T, Levy M, Sadan B, Shalit M, Geisslinger G, Brune K. Computerized surveillance of adverse drug reactions in hospital: pilot study. Br J Clin Pharmacol 1998; 45 (03) 309-314
• 18 Tse CS, Madura AJ. An adverse drug reaction reporting program in a community hospital. QRB Qual Rev Bull 1988; 14 (11) 336-340
• 19 Dalton-Bunnow MF, Halvachs FJ. Computer-assisted use of tracer antidote drugs to increase detection of adverse drug reactions: a retrospective and concurrent trial. Hosp Pharm 1993; 28 (08) 746-749
• 20 Levy M, Azaz-Livshits T, Sadan B, Shalit M, Geisslinger G, Brune K. Computerized surveillance of adverse drug reactions in hospital: implementation. Eur J Clin Pharmacol 1999; 54 (11) 887-892
• 21 Tolley CL, Slight SP, Husband AK, Watson N, Bates DW. Improving medication-related clinical decision support. Am J Health Syst Pharm 2018; 75 (04) 239-246
• 22 Phansalkar S, Zachariah M, Seidling HM, Mendes C, Volk L, Bates DW. Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc 2014; 21 (e2): e332-e340
• 23 Schiff GD, Klass D, Peterson J, Shah G, Bates DW. Linking laboratory and pharmacy: opportunities for reducing errors and improving care. Arch Intern Med 2003; 163 (08) 893-900
• 24 Classen DC, Pestotnik SL, Evans RS, Burke JP. Computerized surveillance of adverse drug events in hospital patients. JAMA 1991; 266 (20) 2847-2851
• 25 Honigman B, Lee J, Rothschild J. , et al. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc 2001; 8 (03) 254-266
• 26 Jha AK, Kuperman GJ, Teich JM. , et al. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998; 5 (03) 305-314
• 27 Kane-Gill SL, Dasta JF, Schneider PJ, Cook CH. Monitoring abnormal laboratory values as antecedents to drug-induced injury. J Trauma 2005; 59 (06) 1457-1462
• 28 Schiff GD, Aggarwal HC, Kumar S, McNutt RA. Prescribing potassium despite hyperkalemia: medication errors uncovered by linking laboratory and pharmacy information systems. Am J Med 2000; 109 (06) 494-497
• 29 Szekendi MK, Sullivan C, Bobb A. , et al. Active surveillance using electronic triggers to detect adverse events in hospitalized patients. Qual Saf Health Care 2006; 15 (03) 184-190
• 30 Silverman JB, Stapinski CD, Huber C, Ghandi TK, Churchill WW. Computer-based system for preventing adverse drug events. Am J Health Syst Pharm 2004; 61 (15) 1599-1603
• 31 Wong A, Wright A, Seger DL, Amato MG, Fiskio JM, Bates D. Comparison of overridden medication-related clinical decision support in the intensive care unit between a commercial system and a legacy system. Appl Clin Inform 2017; 8 (03) 866-879
• 32 Awdishu L, Coates CR, Lyddane A. , et al. The impact of real-time alerting on appropriate prescribing in kidney disease: a cluster randomized controlled trial. J Am Med Inform Assoc 2016; 23 (03) 609-616
• 33 Tawadrous D, Shariff SZ, Haynes RB, Iansavichus AV, Jain AK, Garg AX. Use of clinical decision support systems for kidney-related drug prescribing: a systematic review. Am J Kidney Dis 2011; 58 (06) 903-914
• 34 Jha AK, Laguette J, Seger A, Bates DW. Can surveillance systems identify and avert adverse drug events? A prospective evaluation of a commercial application. J Am Med Inform Assoc 2008; 15 (05) 647-653
• 35 Kilbridge PMAL, Ahmad A. Implementation of a system for computerized adverse drug event surveillance and intervention at an Academic Medical Center. J Clin Outcomes Manag 2006; 13 (02) 94-100
• 36 Nanji KC, Slight SP, Seger DL. , et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014; 21 (03) 487-491
• 37 Rommers MK, Zwaveling J, Guchelaar HJ, Teepe-Twiss IM. Evaluation of rule effectiveness and positive predictive value of clinical rules in a Dutch clinical decision support system in daily hospital pharmacy practice. Artif Intell Med 2013; 59 (01) 15-21
• 38 Hospitalaria SEdF. 2020 Hacia el futuro con seguridad. Madrid: 2015
• 39 Kane-Gill SL, MacLasco AM, Saul MI. , et al. Use of text searching for trigger words in medical records to identify adverse drug reactions within an intensive care unit discharge summary. Appl Clin Inform 2016; 7 (03) 660-671
• 40 Boyce RD, Jao J, Miller T, Kane-Gill SL. Automated screening of emergency department notes for drug-associated bleeding adverse events occurring in older adults. Appl Clin Inform 2017; 8 (04) 1022-1030
• 41 Nightingale PG, Adu D, Richards NT, Peters M. Implementation of rules based computerised bedside prescribing and administration: intervention study. BMJ 2000; 320 (7237): 750-753
• 42 Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003; 348 (25) 2526-2534