Validation of an Algorithm to Classify Urine Cultures in Family Medicine

Abstract

Objectives Automation of test follow-up offers potential reductions in workload for clinicians. The primary objective of the study was to evaluate the performance of MicrobEx, a regular expression-based algorithm in classifying urine culture reports in primary care.

Methods A retrospective validation of MicrobEx was performed using urine culture reports abstracted from a single academic family health team. MicrobEx classifications were compared with labels assigned manually by a human reviewer. Measures of diagnostic performance were calculated.

Results MicrobEx achieved 95.3% accuracy, 88.6% sensitivity, and 100% specificity in classifying 1,999 urine culture reports.

Conclusion The accuracy of MicrobEx was comparable to its performance in the original development and validation study by Eickelberg. Additional work is required to explore and improve the accuracy of MicrobEx and assess its performance across primary care settings and with more complex urine culture reports.

Keywords

urinary tract infections - natural language processing - electronic medical records - family medicine

Protection of Human and Animal Subjects

This study received research ethics board approval.

Publication History

Received: 15 July 2024

Accepted: 09 December 2024

Article published online:
23 April 2025

• References

• 1 Mikhaeil JS, Jalali H, Orchanian-Cheff A, Chartier LB. Quality assurance processes ensuring appropriate follow-up of test results pending at discharge in emergency departments: a systematic review. Ann Emerg Med 2020; 76 (05) 659-674
• 2 Rajaram A, Clapham G, Nemnom M, Thiruganasambandamoorthy V. CAEP/ACMU 2022 Scientific Abstracts. Can J Emerg Med 2022; 24 (01) 1-100
• 3 Luz CF, Vollmer M, Decruyenaere J, Nijsten MW, Glasner C, Sinha B. Machine learning in infection management using routine electronic health records: tools, techniques, and reporting of future technologies. Clin Microbiol Infect 2020; 26 (10) 1291-1299
• 4 Rockenschaub P, Gill MJ, McNulty D, Carroll O, Freemantle N, Shallcross L. Can the application of machine learning to electronic health records guide antibiotic prescribing decisions for suspected urinary tract infection in the Emergency Department?. PLoS Digit Health 2023; 2 (06) e0000261
• 5 Menon S, Murphy DR, Singh H, Meyer AN, Sittig DF. Workarounds and test results follow-up in electronic health record-based primary care. Appl Clin Inform 2016; 7 (02) 543-559
• 6 de Vries S, Ten Doesschate T, Totté JEE. et al. A semi-supervised decision support system to facilitate antibiotic stewardship for urinary tract infections. Comput Biol Med 2022; 146: 105621
• 7 Kroth PJ, Morioka-Douglas N, Veres S. et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open 2019; 2 (08) e199609
• 8 Flood CM, Thomas B, McGibbon E. Canada's primary care crisis: Federal government response. Healthc Manage Forum 2023; 36 (05) 327-332
• 9 Soong C, Shojania KG. Education as a low-value improvement intervention: often necessary but rarely sufficient. BMJ Qual Saf 2020; 29: 353-357
• 10 Berlyand Y, Raja AS, Dorner SC. et al. How artificial intelligence could transform emergency department operations. Am J Emerg Med 2018; 36 (08) 1515-1517
• 11 Matheny ME, Fitzhenry F, Speroff T. et al. Detection of blood culture bacterial contamination using natural language processing. AMIA Annu Symp Proc 2009; 2009: 411-415
• 12 Eickelberg G, Luo Y, Sanchez-Pinto LN. Development and validation of MicrobEx: an open-source package for microbiology culture concept extraction. JAMIA Open 2022; 5 (02) ooac026
• 13 WELL Health Technologies Corp. . OSCAR Pro. 2024 . Accessed December 16, 2024 at: https://well.company/oscar-pro
• 14 Jones M, DuVall SL, Spuhl J, Samore MH, Nielson C, Rubin M. Identification of methicillin-resistant Staphylococcus aureus within the nation's Veterans Affairs medical centers using natural language processing. BMC Med Inform Decis Mak 2012; 12 (01) 34
• 15 Naik N, Talyshinskii A, Shetty DK, Hameed BMZ, Zhankina R, Somani BK. Smart diagnosis of urinary tract infections: is artificial intelligence the fast-lane solution?. Curr Urol Rep 2024; 25 (01) 37-47