Caries Risk Documentation And Prevention: eMeasures For Dental Electronic Health Records

Abstract

Background Longitudinal patient level data available in the electronic health record (EHR) allows for the development, implementation, and validations of dental quality measures (eMeasures).

Objective We report the feasibility and validity of implementing two eMeasures. The eMeasures determined the proportion of patients receiving a caries risk assessment (eCRA) and corresponding appropriate risk-based preventative treatments for patients at elevated risk of caries (appropriateness of care [eAoC]) in two academic institutions and one accountable care organization, in the 2019 reporting year.

Methods Both eMeasures define the numerator and denominator beginning at the patient level, populations' specifications, and validated the automated queries. For eCRA, patients who completed a comprehensive or periodic oral evaluation formed the denominator, and patients of any age who received a CRA formed the numerator. The eAoC evaluated the proportion of patients at elevated caries risk who received the corresponding appropriate risk-based preventative treatments.

Results EHR automated queries identified in three sites 269,536 patients who met the inclusion criteria for receiving a CRA. The overall proportion of patients who received a CRA was 94.4% (eCRA). In eAoC, patients at elevated caries risk levels (moderate, high, or extreme) received fluoride preventive treatment ranging from 56 to 93.8%. For patients at high and extreme risk, antimicrobials were prescribed more frequently site 3 (80.6%) than sites 2 (16.7%) and 1 (2.9%).

Conclusion Patient-level data available in the EHRs can be used to implement process-of-care dental eCRA and AoC, eAoC measures identify gaps in clinical practice. EHR-based measures can be useful in improving delivery of evidence-based preventative treatments to reduce risk, prevent tooth decay, and improve oral health.

Keywords

dental quality measure - caries risk assessment - electronic health records - caries treatment

Background and Significance

Health care is undergoing a transformation to provide better health, better care, lower cost, and an engaged workforce.[1] [2] Continuous quality improvement is advocated for an ideal health care system to monitor care.[3] Each year in the United States, over 195,000 active dental practitioners provide care to more than 127 million patients, expending billions annually.[4] Quality measures applied to dentistry provide an understanding of the return in terms of quality and value within the health care system.[5] [6]

Tooth decay is a disease affecting 60 to 90% of children and up to 100% of adults worldwide.[7] Tooth decay is a complex and multifactorial disease resulting from an imbalance of protective and risk factors in the oral cavity and it remains the most prevalent chronic disease in the United States, affecting 91% of adults ages 20 to 64, with 27% having untreated tooth decay.[8] Healthy People 2030 goals for adults include reducing untreated decay and tooth loss.[9] Caries management has evolved from a traditional restorative, “drill-and-fill,” surgical intervention to a multilevel approach with assessment and determination of caries risk levels divided into low, moderate, high, and extreme.[10] [11] Caries risk assessment (CRA) is conducted during a clinical examination using a form inclusive of the information on disease indicators, biological, and protective factors to predict the likelihood of future caries development with a goal to determine a personalized caries management plan based on the risk status.[10] [12]

In the last 20 years, the dental community has embraced the philosophy of CRA and management,[13] but our understanding to which extent dental institutions, that is, academic, private, and group practices perform CRAs, and deliver appropriate risk-based caries treatments to patients is limited. With structured patient-level data in electronic health records (EHRs), rather than relying on administrative claims data, it is now possible to use clinical quality measures to evaluate our patients' oral health care. A definition of the measure domain process of care is “a health care–related activity performed for, on behalf of, or by a patient,” and “supported by evidence that the clinical process has led to improved outcomes.”[14] The literature reports on the development and implementation of process of care clinical quality measures seeking to improve fluoride applications, oral evaluations for patients with diabetes,[15] tobacco use and cessation,[16] and placement of dental sealants.[17] [18] In 2020, the Dental Quality Alliance released technical specifications of a process of care measure to evaluate caries risk documentation using program/plan administrative enrollment and claims data in children aged 21 years and younger,[19] combining primary, mixed, and permanent dentitions. Our study builds on the Dental Quality Alliance measure, utilizing EHR derived data.

Objectives

In this research study, we employed a process of care quality measurement to evaluate caries risk documentation in children and adults in clinical practice, and evaluated if patients at elevated risk received evidence-based caries preventive management care. Our research objectives are mentioned below:

1. eCRA: develop and validate a CRA measure to determine the percentage of patients (any age) of record who received a CRA within the reporting year.

2. Appropriateness of care eMeasure: develop and validate a risk-based measure to determine the percentage of patients (any age) who received a CRA within the reporting year and corresponding appropriate risk-based treatment(s) on or after the date of the CRA during the reporting year.

Methods

The development, implementation, and validation of eMeasures eCRA and eMeasure appropriateness of care (eAoC) were performed by our team with expertise in caries risk documentation and management, general dentistry, public health, dental informatics, and dental diagnosis terminology. Using retrospective patient-level data, we implemented the research study in three dental schools and one large dental accountable care organization (ACO). All four institutions use the axiUm EHR platform (Exan Corp, Coquitlam, BC, Canada) and the Standardized Diagnostic Dental terminology (SNODDS).[20] Each participating institution obtained respective Institutional Review Board (IRB) approval. As a first step, a query was developed and validated as described earlier.[15] Query validation was conducted in four dental institutions. One institution (site 4) had less than 25 charts with documentation of caries risk, therefore the participation of site 4 in this research study was limited to quality measure validation.

Dental Quality Measures eMeasure Carries Risk Assessment and eMeasure Appropriateness of Care: Development, Implementation, and Validation

[Fig. 1] includes a flowchart with the eMeasures description, the numerator, and denominator details.

Fig. 1 Flow chart with description of caries risk assessment and appropriateness of care eMeasures (eCRA and eAoC, respectively).

eCRA—numerator: we defined the numerator population based on the latest documented caries risk levels in the reporting calendar year of 2019 with three approaches following a hierarchy of events, A1, A2, and A3.

A1. Caries risk diagnosis SNODDS—treatment valid pairs: a caries risk level–related SNODDS[20] diagnosis term (extreme, high, moderate, or low) associated with a planned, in process, or completed valid procedure code, recorded in the EHR. For example, caries risk high SNODDS term (596479), paired with topical fluoride varnish treatment code (D1206), is a valid pair because of a topical fluoride varnish application is an evidence-based preventive treatment indicated for a high-risk caries patient. [Supplementary Appendix A] (available in the online version) details the complete list of caries risk diagnosis-treatment valid pairs.

A2. CRA procedure codes: we followed the Dental Quality Alliance released specifications for practice-level measures for dental clinics to evaluate caries risk documentation using procedure codes[21]; D0601 (caries risk low), D0602 (caries risk moderate), or D0603 (caries risk high) recorded in the patient chart with a “completed” status.

A3. CRA forms: while the Caries Management by Risk Assessment (CAMBRA) form[10] is in use across all participating institutions, we identified the various CRA forms used at each institution, and included those in the study to capture all risk assessment evaluations.

Patient-level caries risk documentation can happen more than once within the reporting year and the CRA may be recorded with any or all of the methods. To address this scenario, we utilized a three-step hierarchy of events for the classification of caries risk levels: (A1) caries risk diagnosis SNODDS–treatment valid pairs; (A2) CRA procedure codes; and (A3) CRA forms. We calculated the measure rate as the ratio of the numerator divided by the denominator.

eAoC is a measure of the caries preventive management plan that is risk-based with patients at elevated risk (moderate, high, or extreme) receiving evidence-based therapies. The consensus recommendation is prescription fluoride for patients at elevated risk levels, and antimicrobials treatments for high and extreme caries risk patients.[10] All participating institutions follow the evidence-based CAMBRA protocol for caries management and prevention which includes prescription 1.1% fluoride toothpaste, professionally applied fluoride (D1206 or D1208), and antimicrobials, or prescription chlorhexidine gluconate 0.12%. We also included the application of interim caries arresting medication (D1354) as fluoride therapy. We excluded low-caries risk patients from the numerator because they do not have active disease and the recommendation is to continue oral hygiene with over the counter toothpaste and regular dental visits.[10]

Development of Automated Electronic Health Record Query

A structured query language (SQL) script was developed to enable standardized extraction of EHR data across participating institutions. During this test, we confirmed that the information extracted by the query from patient charts was 100% accurate and developed the final script for implementation at each site.

Validating the Automated Query

After implementing the script, we validated the query performance with manual chart reviews following our previously described process.[22]

Specifically the below-mentioned steps:

• Step 1: sample size calculations: the queried sample was the population, the respective proportions (measure scores) by site, and we used standard two-tailed z-score of 1.96 and a margin for error of 0.05 (d = 0.05).[23] The resulting sample sizes for evaluating CRA (n₁) and risk-based treatments (n₂) for each site are: site 1, n₁ = 314 and n_{2 =} 362; site 2, n₁ = 326 and n₂ = 317; and site 3 n₁ = 347and n₂ = 385.

• Step 2: manual chart reviews: at each site, two independent reviewers manually reviewed 50 charts; Cohen's kappa[24] values for interrater reliability were κ = 1.0 for all sites, allowing for a single reviewer to complete the remaining chart reviews at each site.

• Step 3: validity analysis: to evaluate the concordance between the automated query and manual chart reviews, we calculated sensitivity, specificity, and positive and negative predictive values. The overall performance was above 90% for eCRA and eAoC ([Table 1]).

• Step 4: measure scores: calculated as the ratio of the numerator divided by the denominator for automated and manual queries. We assessed differences between the automated and manual queries by using a parametric, one sample z-test for proportions. For analysis purposes, we combined high and extreme caries risk patients, because few patients classified as extreme caries risk: 1, 91, and 590 for sites 1, 2, and 3, respectively; and the dental procedure codes do not differentiate between high and extreme caries risk.

Table 1 Overall validity analysis of eCRA and eAoC measures

Query vs. manual	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
eCRA	90.2	100	100	90
eAoC
Fluorides	90.4	100	100	90.2
Antimicrobials	100	100	100	100
Sensitivity	When the manual review confirmed that the automated query correctly identified the patient in the numerator
Specificity	When the manual review confirmed that the patient, as identified by the automated query was not a part of the numerator
PPV	When the automated query confirmed that the patient was part of the numerator as identified by the manual reviewer
NPV	When the automated query confirmed that the patient was only a part of the denominator as identified by the manual reviewer

Abbreviations: eAoC, eMeasures appropriateness of care; eCRA, eMeasures carries risk assessment; NPV, negative predictive value; PPV, positive predictive value;

Results

eCRA: across all three sites, 269,536 patients met the inclusion criteria for the denominator. The proportion of patients who received a CRA (numerator) was 94.43% (n = 254,510) across all sites. The number of female patients in the numerator and denominator was higher compared with males, with 95% of 150,273 female patients receiving a CRA ([Table 2]). Using a Chi-squared test for homogeneity of proportions, there were statistically significant variations in the measure scores by gender (χ ² = 210.01, p < 0.0001). We stratified our data in four age groups as follows: 0 to 5 (primary dentition), 6 to 12 (mixed dentition), 13 to 18 (permanent dentition), and 18+ (permanent dentition including wisdom teeth, 3rd molars; [Table 2]). Across all sites, the majority of our denominator patient population were adults aged 18 years and older with 96.5% of 269,536 receiving a CRA and the lowest were children aged 0 to 5 years with 70.1% with a CRA. The measure scores were statistically significant by age group (χ ² = 19,838, p < 0.0001). Across all sites, the patient population was racial and ethnically diverse. Whites corresponded to 57.2% of the total denominator population. The measure scores were statistically significant by race with the highest proportion of Native Americans (98.8%) received a CRA and 82.5% of patients in the “Other” race category. The race data needs to be interpreted with caution because of the number of unknowns with 17.6% of the total study population. The measure scores for ethnicity were significant with higher proportion of patients being non-Hispanics (97.7% of 220,785 patients; [Table 2]).

Table 2 Percentage of patients with eCRA and eAoC treatment interventions based on gender, age group, race, and ethnicity

	Population distribution		eCRA		eAoC
Variable	eCRA denominator	Perc. (%)	eCRA numerator	Measure score (%)	eAoC denominator	Total patients receiving fluoride treatment interventions (eAoC numerator)	Measure score (%)	Total patients receiving antimicrobials (eAoC numerator)	Measure score (%)
Total patients	269,536	100	254,510	94.4	128,898	109,836	85.2	47,982	37.2
Gender
Male	118,956	44.1	111,485	93.7	59,569	50,937	85.5	23,330	39.2
Female	150,273	55.8	142,773	95.0	69,153	58,753	85.0	24,576	35.5
Transgender	218	0.1	215	98.6	–	–	–	–	–
Unknown	67	0.0	17	25.4	28	15	53.6	6	21.4
Chi-squared (df = 1)				210.01			7.59		179.93
p-Value				<0.0001			0.0058		<0.0001
Age group (y)
0 to 5	16,042	6.0	11,250	70.1	5,371	4,283	79.7	412	7.7
6 to 12	35,248	13.1	32,767	93.0	20,075	18,346	91.4	4,294	21.4
13 to 18	29,436	10.9	28,284	96.1	15,978	14,393	90.1	4,720	29.5
18+	188,810	70.1	182,209	96.5	87,474	72,814	83.2	38,556	44.1
Chi-squared (df = 3)				19,838			1305.2		6323.1
p-Value				<0.0001			<0.0001		<0.0001
Race
White	154,853	57.2	151,225	97.7	71,095	61,658	86.7	28,656	40.3
African American	9,012	3.3	8,131	90.2	5,294	4,176	78.9	1,695	32.0
Asian	14,250	5.3	12,555	88.1	5,798	4,710	81.2	1,660	28.6
Native American	1,031	0.4	1,019	98.8	586	509	86.9	287	49.0
Pacific Islander	1,185	0.4	1,152	97.2	687	626	91.1	323	47.0
Other	3,508	1.3	2,895	82.5	1,779	1,478	83.1	613	34.5
Multiracial	13,510	5.0	13,067	96.7	7,011	6,306	89.9	2,581	36.8
Unknown (blanks)	47,635	17.6	41,856	87.9	22,613	18,833	83.3	7,966	35.2
Chi-squared (df = 7)				10,506			688		596.77
p-Value				<0.0001			<0.0001		<0.0001
Ethnicity
Hispanic	27,875	11	26,025	93.4	15,066	13,233	87.83	5,186	34.42
Non-Hispanic	220,785	89	215,646	97.7	102,060	89,645	87.84	41,766	40.92
Other	221	0	112	50.7	106	71	66.98	–	–
Chi-squared (df = 2, 3)				3,382.8			7011.4		230.75
p-Value				<0.0001			<0.0001		<0.0001

Abbreviations: eAoC, eMeasures appropriateness of care; eCRA, eMeasures carries risk assessment.

We determined caries risk levels with three strategies. [Fig. 2] illustrates the combined use of approaches (A1) caries risk diagnosis SNODDS—treatment valid pairs; (A2) CRA Current Dental Terminology (CDT) codes; and (A3) CRA forms by site. In site 1, approach, A3 alone represented 23.0% of the caries risk documentation. A combination of approaches A2 and A3 accounted for 60.0% of the caries risk documentation, indicating the caries risk status was determined with a combination of CRA forms and CDT codes ([Fig. 3]). In site 2, a combination of A1 and A3 corresponded to 45.0% of the caries risk estimation. Lastly, in site 3, 98.4% of the caries risk document was done with a combination of A1, A2, and A3 approaches. In summary, site 3 used a combination of the A1, A2, and A3 strategies, while site 1 used more often a combination of A2 and A3, and in site 2, A1 and A3 were more common.

Fig. 2 Three approaches to determine the caries risk level (eCRA measure). eCRA, eMeasures carries risk assessment; SNODDS, Standardized Diagnostic Dental terminology.

Fig. 3 Percentage of patients with fluoride therapies and antimicrobial interventions with extreme/high and moderate risk across three sites.

eAoC: [Table 2] also displays the proportion of patients at elevated (extreme, high, or moderate) caries risk levels who received the appropriate caries risk management interventions, fluoride therapies and antimicrobial interventions. Overall, 85.2% of 128,898 patients with elevated risk received fluoride therapy. Measure scores for fluoride delivery were not statistically significant by gender (χ ² = 7.59, p = 0.0058) as both females and males received the treatment intervention equally. Whereas, antimicrobial intervention was more frequent in males (39.2%) than females (35.5%), and the scores were significant. Also, among 91.4% of elevated risk patients, In the 6 to 12 years of age group received a fluoride intervention and it was the lowest in the 0 to 5 years of age group with 79.7% of patients. The measure scores were statistically significant with antimicrobial intervention in the age group category with 44.1% of adult patients (χ ² = 6323.1, p < 0.0001). In terms of race and ethnicity, the highest proportion of Pacific Islanders received a fluoride intervention (91.1%) and only 28.6% of Asians received antimicrobials. Hispanics and non-Hispanics both equally received fluoride therapies (χ ² = 7,011.4, p < 0.0001; [Table 2]). [Fig. 3] shows the proportion of patients at extreme, high, and moderate levels who received fluoride therapies and antimicrobial interventions. Among high and extreme caries risk patients, the most common treatment intervention were fluorides in site 1 (56%), site 2 (71.1%), and site 3 (93.8%). Also, 80.6% of patients in site 3 in the extreme and high risk categories received antimicrobials, whereas only 2.9% of them received in site 1.

Discussion

Dentistry, as part of the health care system, is embracing quality measures and advocating increased use of EHR data,[25] including interoperability between dentistry and medicine[26] [27] and the development of research and clinical networks to share data.[28] [29] Using dental informatics, the aggregation and sharing of data allow for electronic mining of patient care data beyond claims and administrative data for research, teaching, and clinical quality of care purposes. Electronic dental patient care data are used for usability testing,[30] adherence to clinical practice guidelines[31] [32] as reported in this paper, clinic decision support,[33] treatment planning,[34] education,[35] quality metrics,[22] adverse events,[36] and patient safety. Electronically derived dental quality of care measures includes caries risk and appropriate care, as discussed in this paper, also include a wide variety of dental applications such as sealants,[18] [37] tobacco cessation,[16] dental patients with diabetes, and pregnancy.[38] A critical aspect of the EHR is a robust dental diagnostic interface terminology[39] for providers to select diagnosis and pair the diagnosis to the procedure performed.[40] [41] A robust EHR with dental diagnosis, clinic decision support, and electronic quality measures will allow for patient care data to be used for research, patient care, and education purposes including the expansion of quality improvement in dentistry.[27]

We developed, implemented, and validated two practice-level eMeasures, the eCRA a process of care measure for caries risk documentation and the eAoC, AoC measure for the delivery of evidence-based preventive treatments according to caries risk status in two academic institutions and one dental ACO.

The philosophy of CRA is established in dentistry.[42] There is a lack of standardization in determining caries risk status with different forms being utilized across the profession, with differing levels of accuracy in determining caries risk.[12] [43] [44] [45] The electronic measures developed and validated in this study can be utilized, irrespective of which CRA tool is used to determine caries risk. The important aspect of the eCRA and eAoC measures is that caries risk for the patient is made by the provider. Recent review papers have summarized the current status of CRA and prevention recommendations.[46] [47] There many CRA tools available and used in research and clinical practice to assess risk in children and adults and many preventive intervention care recommendations.[10] [48] [49] Despite the variety of forms available, validation studies and risk prediction models of these forms are limited in their ability to accurately predict future tooth decay in some versions of CRA tools.[13] [42] Validation studies with retrospective study designs conducted in academic settings established that past caries experience is the most critical predictor of future caries experience,[11] [12] with high caries risk patients more likely to develop new caries lesions during the follow-up period. A randomized prospective clinical trial using the CAMBRA demonstrated reduction in tooth decay when caries risk was used to determine and deliver prevention in a systematic way.[50] This method of care was enhanced and implemented on a large scale which demonstrated CRA and standardized preventive treatments lowered new decay well below the Healthy People 2020 baseline and target levels.[51] Evidence-based guidelines including the American Dental Association[52] and the CAMBRA protocol recommend that individuals at lower caries risk continue the traditional oral hygiene regimen, consisting of over the counter fluoride toothpaste and attending regular dental visits. For patients with moderate, high, or extreme caries risk, the personalized caries management plan based on risk includes a combination of oral hygiene and dietary instructions, prescription fluoride therapies, and antimicrobials.[10]

The reimbursement model in dentistry has procedure codes for CRA and documentation; however, no documentation is required for payment which accounts for the lack of standardization. This study utilized the intent of the Dental Quality Alliance measure which is limited in using administrative enrollment and claims data. This type of data are all that is available for payors and plan administrators. Our measures built on the DQA measure to be more provider centric and informative from a quality improvement perspective. One of the significant challenges encountered while evaluating caries risk documentation in this study's participating institutions was the heterogeneity in the usage of CRA tools. In this study, we developed quality measures derived from EHRs that can be applied to any CRA tool utilized. Looking solely in the electronic health record for a completed CRA form can be problematic as data elements may be spread throughout the EHR, from different software systems and different implementations, compromising the effectiveness of dental quality measurement and may lead to underrepresentation of the relevant population (in the numerator). To overcome this challenge, and capture the most complete and accurate information, we used three distinct approaches described in the methodology, diagnosis-treatment pair (A1), CDT codes (A2), and CRA forms (A3) to classify caries risk status. This method expands the applicability of the quality measures as we were successful at the three institutions that participated. For all sites, we were able to use the novel risk assessment strategy “diagnosis and treatment valid pairs (A1),” where each pair is comprised of the risk-level assignment and its corresponding treatment choice. In earlier work, we have documented the development of diagnosis–treatment pair strategy,[53] and the importance of using diagnostic terms in assessing adherence to treatment protocols.[39] To measure quality effectively using EHR data and ultimately improve the care, our patients received complete, accurate, and high-quality data that were available for quality measurement evaluations and research. We anticipate that the approach will be successful at a wide range of dental clinics with differing software and implementations, although customization of the measures would be required.

There were site variations in the measure scores that merit some explanation. Site 3, a dental ACO, had the highest proportion of patients with an eCRA relative to the other sites ([Table 2]). Also, site 3 had higher scores for eAoC compared with sites 1 and 2, with the majority of patients at moderate, high and elevated caries risk receiving appropriate caries management interventions. Variations across sites may be partially explained by the differences in the distribution of at-risk patients and the ACO providers are continuously trained and received feedback on their performance regarding the use of diagnostic terminology and corresponding procedure codes.

Our study employed a process of care measure that can identify best practices and opportunities for quality improvement. The participating dental ACO in this study has such processes in place and the results clearly demonstrate improved delivery of caries management strategies to their high-risk patient population. Moving from measuring quality with data to implementing policies for sustained quality improvement takes a culture shift and leadership support as we have described earlier.[54] We expect that well-defined process of care measures for caries risk, eCRA and good AoC measures for caries prevention, eAoC would result in good outcomes of care measures, eAoC for caries. Recently, the literature reported on the development and validation of automated tools to calculate caries indices from data available in the EHR, and can be adapted to implement eAoC measures.[55] The next steps in this research will be to investigate the relationship between eCRA, eAoC for prevention, and eAoC for incidence of new caries lesions.

Early childhood caries (ECC) is the single most common chronic childhood disease.[56] As noted by the CDC, “oral health disparities are profound in the United States.”[57] Although the prevalence of early childhood caries is 37% among all children aged 2 to 8 years, there are great disparities in prevalence between Hispanic (46%) and non-Hispanic Black (44%) children compared with non-Hispanic white children (31%).[58] The prevalence of severe early childhood caries among disadvantaged groups has been reported up to 85%.[59] Clearly, the time has come to implement preventive measures up front to prevent draconian measures later on, for the children entrusted in our care. The ability to evaluate these quality measures including age, gender, and race/ethnicity can be helpful in understanding other factors that might influence the CRA and preventive management of care. A recent quality improvement study showed that children with a documented CRA were more likely to receive preventive services, while those who did not have a CRA had higher number of invasive restorative procedures.[6] We have begun to include other factors, such as socioeconomic status, rural and urban location adding to these measures based on EHR data which will be beneficial in exploring health disparities.[51] [60] Given the prevalence of tooth decay, these measures have tremendous importance in their ability to provide for practitioners, patients, payors, and public health with tools to evaluate the quality of care and appropriateness of treatment provided.

Evidence-based caries risk management strategies for moderate-to-high caries risk patients continues to include sealants, modification in diet, adequate plaque removal, and fluoride toothpaste as important steps for maintaining and/or reversing caries risk and improving the oral health of our patients.[10]

In summary, this work presents the use of process of care dental quality measure using longitudinal patient data available in the EHR to provide feedback on the clinical care related to the documentation of caries risk and risk-appropriate treatment to identify clinical workflows and areas of concern for clinical improvement initiatives. As future directions, we will expand and implement these process-of-care measures in other dental institutions that are a part of the BigMouth Dental Data Repository.[28] The routine adoption of dental quality measures will optimize clinical performance and, ultimately, precise and patient-centered clinical care.

Limitations

Despite obvious advantages of using clinical EHR data to improve clinical outcomes, we encountered several limitations worth noting. First, as process of care measures were limited to evaluate a procedure performed for the patient, we could not assess the reasons why evidence-based preventive care was not prescribed to a larger number of adult patients at elevated caries risk in sites 1 and 2. Also, we could not evaluate if a lower proportion of high-risk individuals in site 3, a dental ACO, is a result of a well-established caries preventive protocol, leadership structure and clinical organization, or because of potential different populations at each institution. Based on caries risk status, we are currently developing outcomes of care measures derived from EHR data to evaluate new and untreated caries lesions. Second, the number of patients with a CRA might have been underestimated in sites 1 and 2, since most were identified from a variety of CRA forms. As quality measures become more common in clinical practice, detailed information connecting disease conditions and dental procedures as available in SNODDS standardized diagnostic terminology,[61] it will be necessary to appropriately capture the care provided and include patients in the numerator. Lastly, our study was conducted using the same EHR (axiUm) across all three sites and calls into question the generalizability level in non-axiUm user institutions. Nevertheless, the methodology and measure specifications we described may be adapted in other institutions using different EHRs, if they are set to the same standardized codes for treatment and diagnosis.

Conclusion

This study demonstrated the use of patient-level EHR data to validate and implement two dental quality eMeasures on CRA and risk-based treatment in a large and diverse cross-sectional sample. Data, available in the EHR allowed for the dental quality eMeasures, developed and implemented in this study utilizing a combination of methods to capture the caries risk, providing the best results in documenting that patients received a CRA and a risk-based caries preventive treatment.

Clinical Relevance Statement

This study's results indicate that quality measurement research using clinical data can be a valuable tool to evaluate the performance of CRA and risk-based treatments. Findings from such evaluations could inform clinicians, researchers, and administrators to design interventions to optimize clinical performance and provide our patients with evidence-based personalized care.

Multiple Choice Questions

1. Which type of dental quality measure was selected to determine the proportion of patients receiving a caries risk assessment and corresponding appropriate risk-based preventative treatments for patients at elevated risk of caries?

   • Process of care

   • Outcomes of care

   • Structural measures

   Correct Answer: The correct answer is option a. Process of care measures can identify gaps in clinical care, and inform the provider, and administrators on what is working well and what needs action.

2. What approaches were used to validate the eMeasures (mark all that apply)?

   • Expert opinion

   • Concordance between manual chart reviews and automated queries

   • Comparison with external dataset

   Correct Answer: The correct answer is option b. The automated query was validated with manual chart reviews in all participating institutions to verify if the query measured what was intended to measure (see Methods).

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

Each participating institution obtained respective Institutional Review Board (IRB) approval.

^* Contributed equally to this article.

• References

• 1 Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood) 2008; 27 (03) 759-769
• 2 Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med 2014; 12 (06) 573-576
• 3 Berwick DM. Continuous improvement as an ideal in health care. N Engl J Med 1989; 320 (01) 53-56
• 4 Munson B, Vujicic M. Supply of dentists in the United States is likely to grow: Health Policy Institute. Available at: http://www.ada.org/?/media/ADA/Science%20and%20Research/HPI/Files/HPIBrief_1014_1.pdf 2014
• 5 Franklin A, Gantela S, Shifarraw S. et al. Dashboard visualizations: Supporting real-time throughput decision-making. J Biomed Inform 2017; 71: 211-221
• 6 Herndon JB, Horton RA, Ruff JC. Findings from a caries risk assessment registry quality improvement initiative in an oral health partnership model. J Public Health Dent 2021
• 7 Kassebaum NJ, Bernabé E, Dahiya M, Bhandari B, Murray CJ, Marcenes W. Global burden of untreated caries: a systematic review and metaregression. J Dent Res 2015; 94 (05) 650-658
• 8 Dye BA, Thornton-Evans G, Li X, Iafolla TJ. Dental caries and sealant prevalence in children and adolescents in the United States, 2011-2012. NCHS Data Brief 2015; (191) 1-8
• 9 U.S. Department of Health and Human Services. Topics & objectives: oral health: oral health of adults: OH-3 reduce the proportion of adults with untreated dental decay: OH-3.1 reduce the proportion of adults aged 35 to 44 with untreated dental decay. Accessed May 16, 2021: https://www.healthypeople.gov/2020/topics-objectives/topic/oral-health/objectives
• 10 Featherstone JD, Domejean-Orliaguet S, Jenson L, Wolff M, Young DA. Caries risk assessment in practice for age 6 through adult. J Calif Dent Assoc 2007; 35 (10) 703–707-710–713
• 11 Doméjean S, White JM, Featherstone JD. Validation of the CDA CAMBRA caries risk assessment–a six-year retrospective study. J Calif Dent Assoc 2011; 39 (10) 709-715
• 12 Chaffee BW, Cheng J, Featherstone JD. Baseline caries risk assessment as a predictor of caries incidence. J Dent 2015; 43 (05) 518-524
• 13 Fontana M, Carrasco-Labra A, Spallek H, Eckert G, Katz B. Improving caries risk prediction modeling: a call for action. J Dent Res 2020; 99 (11) 1215-1220
• 14 National Quality Measures Clearinghouse. NQMC measure domain definitions. Accessed 19 Nov, 2020: https://www.ahrq.gov/gam/summaries/domain-definitions/index.html
• 15 Neumann A, Kalenderian E, Ramoni R. et al. Evaluating quality of dental care among patients with diabetes: adaptation and testing of a dental quality measure in electronic health records. J Am Dent Assoc 2017; 148 (09) 634-643.e1
• 16 Neumann A, Kumar S, Bangar S. et al. Tobacco screening and cessation efforts by dental providers: a quality measure evaluation. J Public Health Dent 2019; 79 (02) 93-101
• 17 Kumar SV, Bangar S, Neumann A. et al. Assessing the validity of existing dental sealant quality measures. J Am Dent Assoc 2018; 149 (09) 756-764.e1
• 18 Kumar SV, Yansane A, Neumann A. et al. Measuring sealant placement in children at the dental practice level. J Am Dent Assoc 2020; 151 (10) 745-754
• 19 Dental Quality Alliance. Caries risk documentation. Accessed November 15, 2020: https://www.ada.org/~/media/ADA/DQA/2020_CariesRiskDocumentation.pdf?la=en
• 20 Kalenderian E, Ramoni RL, White JM. et al. The development of a dental diagnostic terminology. J Dent Educ 2011; 75 (01) 68-76
• 21 Dental quality alliance. Accessed Nov 19, 2020: https://www.ada.org/resources/research/dental-quality-alliance
• 22 Bhardwaj A, Ramoni R, Kalenderian E. et al. Measuring up: Implementing a dental quality measure in the electronic health record context. J Am Dent Assoc 2016; 147 (01) 35-40
• 23 Cochran WG. Sampling Techniques. 3rd ed.. New York, NY: John Wiley & Sons; 1977
• 24 McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012; 22 (03) 276-282
• 25 Schleyer TK, Thyvalikakath TP, Spallek H, Torres-Urquidy MH, Hernandez P, Yuhaniak J. Clinical computing in general dentistry. J Am Med Inform Assoc 2006; 13 (03) 344-352
• 26 Simon L, Obadan-Udoh E, Yansane AI. et al. Improving oral-systemic healthcare through the interoperability of electronic medical and dental records: an exploratory study. Appl Clin Inform 2019; 10 (03) 367-376
• 27 Kalenderian E, Halamka JD, Spallek H. An EHR with teeth. Appl Clin Inform 2016; 7 (02) 425-429
• 28 Walji MF, Kalenderian E, Stark PC. et al. BigMouth: a multi-institutional dental data repository. J Am Med Inform Assoc 2014; 21 (06) 1136-1140
• 29 Stark PC, Kalenderian E, White JM. et al; Consortium for Oral Health-Related Informatics (COHRI). Consortium for oral health-related informatics: improving dental research, education, and treatment. J Dent Educ 2010; 74 (10) 1051-1065
• 30 Walji MF, Kalenderian E, Piotrowski M. et al. Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR. Int J Med Inform 2014; 83 (05) 361-367
• 31 Spallek H, Song M, Polk DE, Bekhuis T, Frantsve-Hawley J, Aravamudhan K. Barriers to implementing evidence-based clinical guidelines: a survey of early adopters. J Evid Based Dent Pract 2010; 10 (04) 195-206
• 32 O'Donnell JA, Modesto A, Oakley M, Polk DE, Valappil B, Spallek H. Sealants and dental caries: insight into dentists' behaviors regarding implementation of clinical practice recommendations. J Am Dent Assoc 2013; 144 (04) e24-e30
• 33 Mertz E, Bolarinwa O, Wides C. et al. Provider attitudes toward the implementation of clinical decision support tools in dental practice. J Evid Based Dent Pract 2015; 15 (04) 152-163
• 34 Tokede O, Walji M, Ramoni R. et al. Treatment planning in dentistry using an electronic health record: implications for undergraduate education. Eur J Dent Educ 2013; 17 (01) e34-e43
• 35 White JM, Jenson LE, Gansky SA. et al. Dental students' clinical experience across three successive curricula at one U.S. dental school. J Dent Educ 2017; 81 (04) 366-377
• 36 Walji MF, Yansane A, Hebballi NB. et al. Finding dental harm to patients through electronic health record-based triggers. JDR Clin Trans Res 2020; 5 (03) 271-277
• 37 Yansane A, Listl S, Dawda D. et al. Increasing value, reducing waste: tailoring the application of dental sealants according to individual caries risk. J Public Health Dent 2020; 80 (Suppl. 02) S8-S16
• 38 Neumann A, Obadan-Udoh E, Bangar S. et al. Number of pregnant women at four dental clinics and the care they received: a dental quality eMeasure evaluation. J Dent Educ 2019; 83 (10) 1158-1165
• 39 Kalenderian E, Ramoni RB, White JM. et al. The importance of using diagnostic codes. Oral Surg Oral Med Oral Pathol Oral Radiol Endod 2011; 112 (01) 4-5
• 40 Tokede O, White J, Stark PC. et al. Assessing use of a standardized dental diagnostic terminology in an electronic health record. J Dent Educ 2013; 77 (01) 24-36
• 41 Yansane A, Tokede O, White J. et al. Utilization and validity of the dental diagnostic system over time in academic and private practice. JDR Clin Trans Res 2019; 4 (02) 143-150
• 42 Cagetti MG, Bontà G, Cocco F, Lingstrom P, Strohmenger L, Campus G. Are standardized caries risk assessment models effective in assessing actual caries status and future caries increment? A systematic review. BMC Oral Health 2018; 18 (01) 123
• 43 Chaffee BW, Cheng J, Featherstone JD. Baseline caries risk assessment and anti-caries therapy predict caries incidence. J Dent Res 2014; 93 (0B): 190
• 44 Walsh CJ, Gansky SA, Vaderhobli RM, Fontana M, Rechmann P, White JM. Usability testing of clinical decision support caries risk assessment tools. J Dent Res 2015; 94 (0A): 1080
• 45 Lin J, Martin J, Yansane A. Reliability testing of caries risk assessment tools with a synthetic database. J Dent Res 2019; 98 (A): 3551
• 46 Featherstone JDB, Crystal YO, Alston P. et al. A comparison of four caries risk assessment methods. Frontiers in Oral Health. 2021; 2 (15) DOI: 10.3389/froh.2021.656558.
• 47 Featherstone JDB, Crystal YO, Alston P. et al. Evidence-based caries management for all ages-practical guidelines. Frontiers in Oral Health. 2021;2(14):
• 48 Young DA, Nový BB, Zeller GG, Hale R, Hart TC, Truelove EL. American Dental Association Council on Scientific Affairs, American Dental Association Council on Scientific Affairs. The American Dental Association Caries Classification System for clinical practice: a report of the American Dental Association Council on Scientific Affairs. J Am Dent Assoc 2015; 146 (02) 79-86
• 49 Ismail AI, Pitts NB, Tellez M. et al; Authors of International Caries Classification and Management System (ICCMS). The International Caries Classification and Management System (ICCMS) an example of a caries management pathway. BMC Oral Health 2015; 15 (Suppl. 01) S9
• 50 Featherstone JD, White JM, Hoover CI. et al. A randomized clinical trial of anticaries therapies targeted according to risk assessment (caries management by risk assessment). Caries Res 2012; 46 (02) 118-129
• 51 White JM, Brandon RG, Mullins JM, Simmons KL, Kottek AM, Mertz EA. Tracking oral health in a standardized, evidence-based, prevention-focused dental care system. J Public Health Dent 2020; 80 (Suppl. 02) S35-S43
• 52 Slayton RL, Urquhart O, Araujo MWB. et al. Evidence-based clinical practice guideline on nonrestorative treatments for carious lesions: a report from the American Dental Association. J Am Dent Assoc 2018; 149 (10) 837-849
• 53 White JM, Kalenderian E, Stark PC, Ramoni RL, Vaderhobli R, Walji MF. Evaluating a dental diagnostic terminology in an electronic health record. J Dent Educ 2011; 75 (05) 605-615
• 54 Ramoni RB, Etolue J, Tokede O. et al. Adoption of dental innovations: The case of a standardized dental diagnostic terminology. J Am Dent Assoc 2017; 148 (05) 319-327
• 55 White JM, Mertz EA, Mullins JM. et al. Developing and testing electronic health record-derived caries indices. Caries Res 2019; 53 (06) 650-658
• 56 Colak H, Dülgergil CT, Dalli M, Hamidi MM. Early childhood caries update: a review of causes, diagnoses, and treatments. J Nat Sci Biol Med 2013; 4 (01) 29-38
• 57 Centers for Disease Control and Prevention. Disparities in oral health. Accessed 20 November, 2020: https://www.cdc.gov/oralhealth/oral_health_disparities/index.htm
• 58 Dye BA. Dental Caries and Sealant Prevalence in Children and Adolescents in the United States, 2011–2012. Hyattsville, MD: National Center for Health Statistics; 2015
• 59 Anil S, Anand PS. Early childhood caries: prevalence, risk factors, and prevention. Front Pediatr 2017; 5: 157
• 60 Cheng J, Gregorich SE, Gansky SA. et al. Constructing matched groups in dental observational health disparity studies for causal effects. JDR Clin Trans Res 2020; 5 (01) 82-91
• 61 SNODDS: dental diagnostic system. Accessed November 21, 2020: http://dentaldiagnosticterminology.org/

Publication History

Received: 18 February 2021

Accepted: 30 October 2021

Article published online:
19 January 2022