Feasibility of Electronic Health Record–Based Triggers in Detecting Dental Adverse Events

Abstract

Background We can now quantify and characterize the harm patients suffer in the dental chair by mining data from electronic health records (EHRs). Most dental institutions currently deploy a random audit of charts using locally developed definitions to identify such patient safety incidents. Instead, selection of patient charts using triggers and assessment through calibrated reviewers may more efficiently identify dental adverse events (AEs).

Objective Our goal was to develop and test EHR-based triggers at four academic institutions and find dental AEs, defined as moderate or severe physical harm due to dental treatment.

Methods We used an iterative and consensus-based process to develop 11 EHR-based triggers to identify dental AEs. Two dental experts at each institution independently reviewed a sample of triggered charts using a common AE definition and classification system. An expert panel provided a second level of review to confirm AEs identified by sites reviewers. We calculated the performance of each trigger and identified strategies for improvement.

Results A total of 100 AEs were identified by 10 of the 11 triggers. In 57% of the cases, pain was the most common AE identified, followed by infection and hard tissue damage. Positive predictive value (PPV) for the triggers ranged from 0 to 0.29. The best performing triggers were those developed to identify infections (PPV = 0.29), allergies (PPV = 0.23), failed implants (PPV = 0.21), and nerve injuries (PPV = 0.19). Most AEs (90%) were categorized as temporary moderate-to-severe harm (E2) and the remainder as permanent moderate-to-severe harm (G2).

Conclusion EHR-based triggers are a promising approach to unearth AEs among dental patients compared with a manual audit of random charts. Data in dental EHRs appear to be sufficiently structured to allow the use of triggers. Pain was the most common AE type followed by infection and hard tissue damage.

Keywords

triggers - adverse events - dentistry - harm - patient safety

Background and Significance

At the turn of the 21st century, two landmark reports from the National Academy of Medicine reignited the country's commitment to patient safety and health care quality by quantifying the adverse events (AEs) exacted upon the patient population.[1] [2] [3] Like medicine, dentistry's exceedingly sophisticated procedures entail conceivable risk to patients,[4] which may ultimately result in harm.[5] The highlighted deficits in patient safety and quality improvement have initiated considerable growth in health care performance evaluation; however, the virtues of that work have not permeated the U.S. dental health care delivery system.[5] Only the most serious of dental AEs are known due to legal cases, case reports,[6] or reports in the news media.

“Measurement forms the basis of evaluation and has become one of the foundations of current efforts to improve health care.”[7] The trigger tool methodology was developed, recognizing that “conventional approaches to identifying and quantifying harm such as individual chart audits, incident reports, or voluntary administrative reporting have often been less successful in improving the detection of adverse events.”[8] In 2003, the Global Trigger Tool (GTT) was developed to measure inpatient AEs using a paper-based format.[9] [10] A systematic review in 2016 showed that “substantial differences in AE rates were evident across studies, most likely associated with methodological differences and disparate reviewer interpretations.”[11] Recently, a broadly applicable pediatric trigger tool was developed to facilitate the identification of AEs in pediatric inpatients.[12] Informed by the development of the medical outpatient trigger tool,[13] the authors have been able to detect nonmortality-related dental AEs using electronic health record (EHR) based triggers.[14] Dentistry as a profession is slowly embracing the patient safety era, with a few dental care researchers focusing on AE reduction.[15] [16] [17]

A “trigger” is an opportunity or clue to identify AEs in a patient's EHR. However, triggers themselves do not represent AEs. Using triggers for targeted retrospective review, though, provides health care providers with crucial information regarding potential safety risks. As such, patient records are a valuable source of data that can help identify AEs. Traditionally, a random sample of health records is selected for audit. However, Classen et al found that a focused chart review identifies more AEs than random chart review.[18] Some medical records may be more likely to contain AEs, hence the use of a targeted trigger. For example, one would use a trigger specifically designed to find medication errors such as the administration of a reversal drug like naloxone if looking for harm due to an overdose. Consequently, it is of the utmost importance to take great care in developing each trigger. Triggers were first developed as paper-based tools,[8] but the increased use of EHRs among large dental institutions makes them a promising approach to more efficiently identify patient harm.[19]

Objective

The objective of this study was to develop EHR-based targeted dental triggers to allow for the identification of AEs from electronic dental records.

Methods

Trigger Development

Inspired by the Institute for Healthcare Improvement (IHI) GTT, we used an iterative process to develop 11 EHR-based triggers to identify dental AEs. [Table 1] shows the triggers developed by a team of experts with collective experience in dentistry, quality improvement, and informatics. Some triggers targeted specific AEs such as aspiration or ingestion of foreign bodies. Others, such as multiple visits, were general purpose triggers that cast a wider net. We used an iterative consensus-based process to develop and finalize the logic for each trigger ([Appendix A]). The triggers relied on both structured and unstructured data from the EHR. All site reviewers used the same EHR, axiUm (Exan, Vancouver, British Columbia, Canada). Structured data included medications, treatment (CDT [Code on Dental Procedures and Nomenclature]) codes,[20] and diagnostic terms (SNODDS[21] [22]) that are neatly encoded in the EHR. When structured data were not available, we used the rich descriptions in the unstructured clinical notes. The triggers were automated queries run against the electronic dental records in a specific calendar year.

Table 1 Definition and classification system used to identify dental AEs

AE definition	Physical harm that is moderate or severe due to treatment within a specific time frame
AE type	1. Allergy/toxicity/foreign body response
	2. Aspiration/ingestion of foreign bodies
	3. Infections
	4. Wrong-site, wrong-procedure, and wrong-patient errors
	5. Bleeding
	6. Pain
	7. Hard tissue injury
	8. Soft tissue injury
	9. Nerve injury
	10. Other systemic complications
	11. Other orofacial complications
	12. Other harm
AE severity	E1: temporary minimal harm E2: temporary moderate-to-severe harm G1: permanent minimal harm G2: permanent moderate-to-severe harm

Abbreviation: AE, adverse event.

Appendix A Trigger logic to identify dental adverse events

Trigger name	Trigger description
Allergy/toxicity/foreign body response	Patients who had “foreign body” text in their notes and had received at least one treatment in a given calendar year
Aspiration/ingestion of foreign bodies	Patients who had terms such as “aspiration” and “aspirated” in their notes and had received at least one treatment in a given calendar year
Extraction following a crown or RCT or filling	Patients who had an extraction in a given calendar year and also had a procedure for crown, RCT, or filling on the same tooth within the last 365 days from the extraction
Failed implant	Patients who had a failed implant diagnosis or implant removal procedure code on any tooth in a given calendar year
Infections	Patients having either surgical extractions or periodontal surgical procedures and (1) received any prescription medications such as antibiotics, steroids, or pain killers (except chlorhexidine) 1 to 7 days after dental treatment or (2) recorded a dental treatment 1 to 7 days after the aforementioned treatments
Multiple visits	Patients who had 12 or more visits in 6 months in a given calendar year
Nerve injury	Patients who had “paresthesia” noted in the clinical notes within 1 to 5 days after receiving treatment in a given calendar year
Post- surgical extraction complications or postperiodontal treatment complications	Patients who received an extraction or a periodontal surgical procedure and then received any (1) prescription medication 1 to 5 days after dental treatment or (2) had an emergency visit or a follow-up visit with complications
Repeated restorations	Patients who received a repeated restoration on the same tooth with an overlap of one surface within the past 2 years Patients with two or more completed fillings on the same tooth with an overlap of one surface in a given calendar year who also received fillings on the same or another tooth within past 2 years
Soft tissue injury or inflammation	Patients who had “laceration,” “ulcer,” “swelling,” or “burn” noted in their clinical notes within 1 to 5 days after receiving treatment in a given calendar year
Untreated periodontitis	Patients having either (1) chronic periodontitis diagnosis or (2) periodontal initial examination in a given calendar year, with 30% of teeth present having clinical attachment loss ≥ 5 mm and the patient has received an FPD/RPD/crown treatment, and (3) no periodontal treatment (SRP) within 12 months of diagnosis

Abbreviations: FPD, fixed partial denture; RCT, root canal treatment; RPD, removable partial denture; SRP, scaling and root planning.

Chart Review Process

Once the automated trigger script identified relevant charts, a sample of charts retrieved by each trigger was independently reviewed by two dental experts at each site using a common AE definition ([Table 1]). At two sites more than 100 charts were triggered; in these instances 50 charts were reviewed at random. Otherwise, all triggered charts were reviewed. The reviewers determined if the trigger was actually present in the chart (to assess the validity of the automated query). The chart was then manually reviewed for any and all AEs within the time frame, regardless of whether the AE was directly related to the trigger or not. When an AE was found, chart reviewers abstracted pertinent information and summarized the case, categorized the type of AE, and assigned a severity score ([Table 1]). [Appendix B] shows the chart abstraction form developed in REDCap.[23] Chart reviewers then met to compare their findings and to finalize the AEs.

Appendix B RedCAP data capture forms.

Expert Panel Review

Chart reviewers from each site and additional senior investigators in the research team formed an expert panel to provide a second level of review for each AE identified by the sites. First, expert panelists independently reviewed each AE case. The group then connected together on conference calls to make a final determination if the case was an AE. The group also determined a category that best described the AE and assigned a severity rating.

Statistical Analysis

A descriptive analysis of the “triggered” charts and AEs detected will be reported. Among the identified triggered charts, a small sample was reviewed to determine the AE status, type, and severity by the 11 calibrated reviewers. A sample of approximately one in four triggered charts was reviewed. Sample sizes were estimated according to the resources and available capacity of the reviewers at the specific institution. Estimation of the agreement among the individual expert panelists was established using the appropriate Kappa coefficient. Before the expert panel group review, data from the individual reviews were compiled, and the percent agreement and the Prevalence and Bias-Adjusted Kappa (PABAK) were estimated. The ratings from all reviewers at each of the sites were included in the kappa calculations, and the PABAK was estimated for all triggers in aggregate. The average percent agreement for AE determination was 81.9% and the PABAK was 55.2% (κ = 0.55) for determining AE presence. The average percent agreement for categorization of the AE type was 79.6%, whereas the PABAK was 48.4%. Lastly, the average percent agreement for categorization of AE severity was 82.8%, and the corresponding PABAK was 69.1%. According to the standards for interrater reliability, κ ranging from 0.40 to 0.60 constitutes moderate agreement ([Table 2]).[24] [25] To evaluate the effectiveness of each trigger, the overall sensitivity was calculated. The positive predictive value (PPV) diagnostic measure was calculated for each trigger along with the 95% confidence intervals. All analysis was performed using R version 3.4.1 (2017, The R Foundation for Statistical Computing).

Table 2 Interrater reliability among reviewers within the expert panel

Category	Percent agreement	PABAK
AE determination	81.8%	55.2%
AE type	79.6%	48.4%
AE severity	82.8%	69.1%

Abbreviations: AE, adverse event; PABAK, Prevalence and Bias-Adjusted Kappa.

Results

As shown in [Table 3], a total of 3,283 charts were triggered after executing the 11 triggers at the four institutions over the 1-year time frame. Of these, a random sample of 859 charts was manually reviewed and a total of 100 AEs were identified ([Table 4]). In our sample of confirmed AEs where we knew the AE type, the triggers were able to detect the appropriate chart containing the intended trigger AE 78.3% (0.68–0.86) of the time, and the false-negative rate of AE detection was 21.8% (0.14–0.32). PPV for the individual triggers ranged from 0 to 0.29. Only the repeated restoration trigger failed to identify any AE. The best performing triggers were those that developed to identify infections with PPV = 0.29 (0.20–0.39), allergy/toxicity/foreign body response with PPV = 0.23 (0.11–0.40), failed implants PPV = 0.21 (0.09–0.38), and nerve injuries with PPV = 0.19 (0.09–0.37).

Table 3 Trigger performance

Trigger name	Charts triggered	Charts reviewed	Charts with AE	Positive predictive value (95% CI)
1. Infections	430	100	29	0.29 (0.20–0.39)
2. Allergy/toxicity/foreign body response	36	35	8	0.23 (0.11–0.40)
3. Failed implant	34	34	7	0.21 (0.09–0.38)
4. Nerve injury	36	36	7	0.19 (0.09–0.37)
5. Postsurgical extraction complications or postperiodontal treatment complications	377	100	16	0.16 (0.09–0.25)
6. Extraction following a crown or RCT or filling	110	99	9	0.09 (0.05–0.17)
7. Soft tissue injury	1449	100	7	0.07 (0.031–0.14)
8. Untreated periodontitis	224	100	7	0.07 (0.03–0.14)
9. Multiple visits	60	58	1	0.017 (0.009–0.10)
10. Aspiration/ingestion of foreign bodies	136	68	1	0.015 (0.007–0.09)
11. Repeated restorations	391	129	0	0 (0–0.29)

Abbreviations: AE, adverse event; CI, confidence interval; RCT, root canal treatment.

Table 4 Classification of dental adverse events

AE categories	AE count	Examples
Pain	57	Severe pain, pain due to dehiscence
Infection	16	Abscess, trismus, dry socket, infection postperiodontal procedure
Hard tissue damage	11	Tooth damage, root canal perforation, bone damage after implant placement
Nerve injury	6	Numbness, paresthesia
Soft tissue injury	5	Necrosis, laceration
Other orofacial complications	2	Facial pain, sinus perforation
Allergy/toxicity/foreign body response	1	Drug allergy
Aspiration/ingestion of foreign bodies	1	Ingestion of foreign bodies such as prosthesis
Other systemic complications	1	Vomiting
Wrong-site, wrong-procedure, wrong-patient errors	0
Bleeding	0
Other harm	0
Total	100

Abbreviation: AE, adverse event.

Each of the identified AEs was classified further ([Tables 4] and [5]). Pain (57%) was the most frequently identified AE type followed by infection (16%) and hard tissue damage (11%). Most AEs (90%) were categorized as temporary moderate-to-severe harm (E2) and the remainder as permanent moderate-to-severe harm (G2). No AEs were categorized as mild, temporary harm (E1), and mild, permanent harm (G1).

Table 5 Severity of adverse events

AE categories	AE count
E2 (temporary moderate-to-severe harm)	90
G2 (permanent moderate-to-severe harm)	10
Total	100

Abbreviation: AE, adverse event.

Discussion

We demonstrated the feasibility of using EHR-based triggers to identify patient records that contain AEs. Using structured EHR data fields, such as CDT procedure codes, prescription data fields, and standardized diagnostic terms, greatly facilitate the development of triggers. With the adoption of the SNODDS terminology as an American National Standards Institute standard, it is our assumption that the use of dental diagnostic terms will become more conventional. We also benefited from the use of a standardized tooth numbering system that helped to better specify trigger logic. Some of our triggers included data from the periodontal chart. Using periodontal charting has proven to be quite complex as a clinical measure; clinical attachment level, one of the periodontal measures, has six values (one for each surface) for each tooth. And lastly, we also resorted to text mining of the clinical notes to identify key words for a few triggers, such as nerve injury trigger. Not surprisingly, the challenge of this approach was the large number of false-positives. For example, just the word “anesthesia” and its wildcard variations (*) were not possible to use as many notes included sentences such as “no permanent anesthesia noted” or “sufficient anesthesia was obtained.” Despite these limitations, some triggers that relied on text mining such as “allergy/toxicity/foreign body response” and “nerve injury” were among the better performing triggers with higher PPVs. In our next phase, we look forward to using natural language processing techniques to further improve the performance of these triggers. Our trigger scripts were developed for the axiUm EHR used by many large academic dental institutions. We expect that the logic of the scripts can be translated to other EHRs including those used by small dental practices. However, there may be challenges due to the lack of consistent data standards and controlled vocabularies currently used in dentistry, and more research is needed to determine their applicability and performance.

Two-stage chart review, while lengthy, provides an avenue to fully discuss cases. At both stages, the individual case reviews included a consensus phase. Hence, by the time our reviewers classified an incident as an AE, multiple perspectives had been provided through various consensus processes. The limitation, of course, is that this process is time-consuming, cumbersome, and thus inherently expensive. However, the process was critical for the team to learn how to apply a common definition for AEs, how to categorize the type of harm, and apply the severity scale. Like the IHI, we have adapted the harm severity rating created by the National Coordinating Council for Medication Error Reporting and Prevention, and for this study, we included only those AEs for which the harm was classified as moderate to severe (E2 or G2). Therefore, minimal harm (E1, G1) was not included. This resulted in lengthy discussions by the expert panel if certain cases were indeed minimal or more moderate harm in nature and should thus be excluded or included. Additionally, there were several cases that fell under the “quality-of-care” rubric and not harm, such as poor chair-site manners. These too led to lengthy discussions. We acknowledge that tracking such quality-of-care events is enormously important as we would not want to lose the ability to study and improve upon them.

Clearly, some EHR-based triggers identify charts with dental AEs better than others. Triggers seeking to identify infections, allergy/toxicity/foreign body response, failed implants, nerve injury, and postsurgery complications had the highest PPVs (0.29–0.16). While the repeated restoration, aspiration/ingestion, and multiple visit trigger performed poorly (PPV = 0.02 to 0), we found that having multiple visits in a short period of time were characteristic of academic dental institutions where trainees require patients to come back multiple times. Our purpose was to determine the potential for these triggers to identify both general and specific AEs, but while the triggers were designed to find specific AEs, in some case they identified other types of harm. In the future work, we will work to revise the triggers and conduct a larger-scale review to determine the performance of these triggers on a statistically appropriate sample.

We found that 57 of the 100 AEs were classified as pain. At the outset of the study, we did not intend to create a separate pain category. We included pain as an AE when it was not just severe, indicated by specific words or sentences such as “kept me up at night,” “10 out of 10,” :unbearable,” and so on, but also when the pain was unexpected or not well managed, that is, the patient returned for an emergency visit. We understand that pain is a complex AE. Patients come to the dentist not always in pain but often will leave with some level of discomfort. There is an understanding that dentistry is “painful.” However, there is school of thought that suggests that all pain should and can be managed and that proper expectations can be set to make sure the patient is well managed during this episode of aftercare.[26] We have started a separate project to further analyze these pain cases and explore this topic more as pain management is paramount to the well-being of our patients and our quest to minimize harm.

The ultimate goal of using triggers is not only to understand the AEs that have happened due to dental procedures but also to move to the next step and perform root cause analyses specifically for instances when we see multiple cases of the same AE. The root cause analysis should allow us to better understand the underlying system issues that need to be addressed to facilitate improvement.

Conclusion

EHR-based triggers are a promising methodology to unearth AEs among dental patients. Using standardized fields in the EHR as part of the trigger logic greatly improves the PPV of the trigger. In this pilot study, pain was the most common AE unearthed, followed by infection and hard tissue damage.

Clinical Relevance Statement

Running triggers against dental clinic records will allow for the detection of harm caused by dental procedures. This, in turn, will provide an opportunity for the clinician to explore underlying system issues to make lasting improvements.

Multiple Choice Questions

1. Please indicate the answer that best describes what a trigger is when used as trigger logic to develop a script to run against an EHR.

   • A trigger is an AE

   • A trigger is an underlying system issue

   • A trigger is a clue to identify AEs in a patient's EHR

   • A trigger represents a safety hazard

   Correct Answer: The correct answer is option c. A “trigger” is an opportunity or clue to identify AEs in a patient's EHR. However, triggers themselves do not represent AEs. Using triggers for targeted retrospective review, though, provides health care providers with crucial information regarding potential safety risks. As such, patient records are a valuable source of data that can help identify AEs.

2. What is the ultimate goal to use EHR-based triggers in your practice?

   • Triggers can identify charts that may hold an AE

   • Triggers are a first step toward understanding why AEs happen

   • Triggers improve the functionality of the EHR

   • Use of triggers will shortly be made mandatory by professional bodies

   Correct Answer: The correct answer is option a. The ultimate goal of using triggers is not only to understand the AEs that have happened due to dental procedures but also to move to the next step and perform root cause analyses specifically for instances when we see multiple cases of the same AE. The root cause analysis should allow us to better understand the underlying system issues that need to be addressed to facilitate improvement.

Conflict of Interest

None.

Note

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Protection of Human and Animal Subjects

This study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by the Institutional Review Board of each participating institution.

• References

• 1 Kohn L, Corrigan J, Donaldson M. , eds; Institute of Medicine Committee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy of Science; 2002
• 2 Richardson WC. , ed; Institute of Medicine Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001
• 3 Leape LL, Berwick DM, Bates DW. What practices will most improve safety? Evidence-based medicine meets patient safety. JAMA 2002; 288 (04) 501-507
• 4 Ramoni RB, Walji MF, White J. , et al. From good to better: toward a patient safety initiative in dentistry. J Am Dent Assoc 2012; 143 (09) 956-960
• 5 Ramoni R, Walji MF, Tavares A. , et al. Open wide: looking into the safety culture of dental school clinics. J Dent Educ 2014; 78 (05) 745-756
• 6 Obadan EM, Ramoni RB, Kalenderian E. Lessons learned from dental patient safety case reports. J Am Dent Assoc 2015; 146 (05) 318-326
• 7 Bhardwaj A, Ramoni R, Kalenderian E. , et al. Measuring up: implementing a dental quality measure in the electronic health record context. J Am Dent Assoc 2016; 147 (01) 35-40
• 8 Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools. Qual Saf Health Care 2003; 12 (Suppl. 02) ii39-ii45
• 9 Naessens JM, O'Byrne TJ, Johnson MG, Vansuch MB, McGlone CM, Huddleston JM. Measuring hospital adverse events: assessing inter-rater reliability and trigger performance of the Global Trigger Tool. Int J Qual Health Care 2010; 22 (04) 266-274
• 10 Classen DC. , et al. Development and evaluation of the Institute for Healthcare Improvement Global Trigger Tool. J Patient Saf 2008; 4 (05) 169-177
• 11 Hibbert PD, Molloy CJ, Hooper TD. , et al. The application of the Global Trigger Tool: a systematic review. Int J Qual Health Care 2016; 28 (06) 640-649
• 12 Unbeck M, Lindemalm S, Nydert P. , et al. Validation of triggers and development of a pediatric trigger tool to identify adverse events. BMC Health Serv Res 2014; 14: 655
• 13 Resar R. IHI Outpatient Adverse Event Trigger Tool. November 4, 2016. Volume 4. Available at: http://www.ihi.org/resources/Pages/Tools/OutpatientAdverseEventTriggerTool.aspx . Accessed January 28, 2018
• 14 Kalenderian E, Walji MF, Tavares A, Ramoni RB. An adverse event trigger tool in dentistry: a new methodology for measuring harm in the dental office. J Am Dent Assoc 2013; 144 (07) 808-814
• 15 Maramaldi P, Walji MF, White J. , et al. How dental team members describe adverse events. J Am Dent Assoc 2016; 147 (10) 803-811
• 16 Perea-Pérez B, Santiago-Sáez A, García-Marín F, Labajo-González E, Villa-Vigil A. Patient safety in dentistry: dental care risk management plan. Med Oral Patol Oral Cir Bucal 2011; 16 (06) e805-e809
• 17 Thusu S, Panesar S, Bedi R. Patient safety in dentistry - state of play as revealed by a national database of errors. Br Dent J 2012; 213 (03) E3-E3
• 18 Classen DC, Resar R, Griffin F. , et al. ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood) 2011; 30 (04) 581-589
• 19 Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates of patient harm resulting from medical care. N Engl J Med 2010; 363 (22) 2124-2134
• 20 American Dental Association. Code on Dental Procedures and Nomenclature (CDT Code). April 17, 2016 2015 . Available at: http://www.ada.org/en/publications/cdt/ . Accessed January 28, 2018
• 21 Kalenderian E, Ramoni RL, White JM. , et al. The development of a dental diagnostic terminology. J Dent Educ 2011; 75 (01) 68-76
• 22 Kalenderian E, Ramoni RB, White JM. , et al. The importance of using diagnostic codes. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 2011; 112 (01) 4-5
• 23 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42 (02) 377-381
• 24 Cunningham M. More than just the kappa coefficient: a program to fully characterize inter-rater reliability between two raters. Paper presented at the SAS Global Forum 2009, Gaylord National Resort and Convention Center, Washington, DC
• 25 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33 (01) 159-174
• 26 Chorney JM, McGrath P, Finley GA. Pain as the neglected adverse event. CMAJ 2010; 182 (07) 732