Improving the Accuracy of a Clinical Decision Support System for Cervical Cancer Screening and Surveillance

Abstract

Background Clinical decision support systems (CDSS) for cervical cancer prevention are generally limited to identifying patients who are overdue for their next routine/next screening, and they do not provide recommendations for follow-up of abnormal results. We previously developed a CDSS to automatically provide follow-up recommendations based on the American Society of Colposcopy and Cervical Pathology (ASCCP) guidelines for women with both previously normal and abnormal test results leveraging information available in the electronic medical record (EMR).

Objective Enhance the CDSS by improving its accuracy and incorporating changes to reflect the latest revision of the guidelines.

Methods After making enhancements to the CDSS, we evaluated the performance of the clinical recommendations on 393 patients selected through stratified sampling from a set of 3,704 patients in a nonclinical setting. We performed chart review of individual patient's record to evaluate the performance of the system. An expert clinician assisted by a resident manually reviewed the recommendation made by the system and verified whether the recommendations were as per the ASCCP guidelines.

Results The recommendation accuracy of the enhanced CDSS improved to 93%, which is a substantial improvement over the 84% reported previously. A detailed analysis of errors is presented in this article. We fixed the errors identified in this evaluation that were amenable to correction to further improve the accuracy of the system. The source code of the updated CDSS is available at https://github.com/ohnlp/MayoNlpPapCdss.

Conclusion We made substantial enhancements to our earlier prototype CDSS with the updated ASCCP guidelines and performed a thorough evaluation in a nonclinical setting to improve the accuracy of the CDSS. The CDSS will be further refined as it is utilized in the practice.

Keywords

clinical decision support system - ambulatory care information systems - testing and evaluation of health information technology - electronic health records - knowledge delivery - knowledge management

Protection of Human and Animal Subjects

This study does not involve any experiments involving human and animal subjects. The institutional review board at the Mayo Clinic, Rochester, MN, approved this study.

Funding

We acknowledge the funding from the Agency for Healthcare Research and Quality (AHRQ), grant number R21H S022911–01, NLP enabled decision support for cervical cancer screening and surveillance that supported this work.

• References

• 1 U.S. Cancer Statistics Working Group. United States Cancer Statistics: 1999–2013 Incidence and Mortality Web-based Report. 2016; Available at: http://www.cdc.gov/uscs
• 2 Duggan MA, Nation J. An audit of the cervical cancer screening histories of 246 women with carcinoma. J Low Genit Tract Dis 2012; 16 (03) 263-270
• 3 Janerich DT, Hadjimichael O, Schwartz PE. , et al. The screening histories of women with invasive cervical cancer, Connecticut. Am J Public Health 1995; 85 (06) 791-794
• 4 Mema SC, Nation J, Yang H. , et al. Screening history in 313 cases of invasive cancer: a retrospective review of cervical cancer screening in Alberta, Canada. J Low Genit Tract Dis 2017; 21 (01) 17-20
• 5 Siegel R, DeSantis C, Virgo K. , et al. Cancer treatment and survivorship statistics, 2012. CA Cancer J Clin 2012; 62 (04) 220-241
• 6 Leyden WA, Manos MM, Geiger AM. , et al. Cervical cancer in women with comprehensive health care access: attributable factors in the screening process. J Natl Cancer Inst 2005; 97 (09) 675-683
• 7 Moyer VA. ; U.S. Preventive Services Task Force. Screening for cervical cancer: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med 2012; 156 (12) 880-891
• 8 Bright TJ, Wong A, Dhurjati R. , et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
• 9 Lobach DF. The road to effective clinical decision support: are we there yet? . BMJ 2013; 346: f1616
• 10 Everett T, Bryant A, Griffin MF, Martin-Hirsch PP, Forbes CA, Jepson RG. Interventions targeted at women to encourage the uptake of cervical screening. Cochrane Database Syst Rev 2011; (05) CD002834
• 11 Kim JJ. Opportunities to improve cervical cancer screening in the United States. Milbank Q 2012; 90 (01) 38-41
• 12 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
• 13 Wagholikar KB, MacLaughlin KL, Henry MR. , et al. Clinical decision support with automated text processing for cervical cancer screening. J Am Med Inform Assoc 2012; 19 (05) 833-839
• 14 Wagholikar KB, MacLaughlin KL, Kastner TM. , et al. Formative evaluation of the accuracy of a clinical decision support system for cervical cancer screening. J Am Med Inform Assoc 2013; 20 (04) 749-757
• 15 Wagholikar KB, MacLaughlin KL, Casey PM. , et al. Automated recommendation for cervical cancer screening and surveillance. Cancer Inform 2014; 13 (Suppl. 03) 1-6
• 16 Wagholikar KB, MacLaughlin KL, Chute CG, Greenes RA, Liu H, Chaudhry R. Granular Quality Reporting for Cervical Cytology Testing. AMIA Jt Summits TranslSci Proc 2015; 2015: 178-182
• 17 Massad LS, Einstein MH, Huh WK. , et al; 2012 ASCCP Consensus Guidelines Conference. 2012 updated consensus guidelines for the management of abnormal cervical cancer screening tests and cancer precursors. ObstetGynecol 2013; 121 (04) 829-846
• 18 Saslow D, Solomon D, Lawson HW. , et al; ACS-ASCCP-ASCP Cervical Cancer Guideline Committee. American Cancer Society, American Society for Colposcopy and Cervical Pathology, and American Society for Clinical Pathology screening guidelines for the prevention and early detection of cervical cancer. CA Cancer J Clin 2012; 62 (03) 147-172
• 19 Schiffman M, Wentzensen N. A suggested approach to simplify and improve cervical screening in the United States. J Low Genit Tract Dis 2016; 20 (01) 1-7
• 20 Davey DD, Cox JT, Austin RM. , et al. Cervical cytology specimen adequacy: patient management guidelines and optimizing specimen collection. J Low Genit Tract Dis 2008; 12 (02) 71-81
• 21 Wagholikar KB, Mandel JC, Klann JG. , et al. SMART-on-FHIR implemented over i2b2. J Am Med Inform Assoc 2017; 24 (02) 398-402