Validation of a Crowdsourcing Methodology for Developing a Knowledge Base of Related Problem-Medication Pairs

Summary

Background: Clinical knowledge bases of problem-medication pairs are necessary for many informatics solutions that improve patient safety, such as clinical summarization. However, developing these knowledge bases can be challenging.

Objective: We sought to validate a previously developed crowdsourcing approach for generating a knowledge base of problem-medication pairs in a large, non-university health care system with a widely used, commercially available electronic health record.

Methods: We first retrieved medications and problems entered in the electronic health record by clinicians during routine care during a six month study period. Following the previously published approach, we calculated the link frequency and link ratio for each pair then identified a threshold cutoff for estimated problem-medication pair appropriateness through clinician review; problem-medication pairs meeting the threshold were included in the resulting knowledge base. We selected 50 medications and their gold standard indications to compare the resulting knowledge base to the pilot knowledge base developed previously and determine its recall and precision.

Results: The resulting knowledge base contained 26,912 pairs, had a recall of 62.3% and a precision of 87.5%, and outperformed the pilot knowledge base containing 11,167 pairs from the previous study, which had a recall of 46.9% and a precision of 83.3%.

Conclusions: We validated the crowdsourcing approach for generating a knowledge base of problem-medication pairs in a large non-university health care system with a widely used, commercially available electronic health record, indicating that the approach may be generalizable across health-care settings and clinical systems. Further research is necessary to better evaluate the knowledge, to compare crowdsourcing with other approaches, and to evaluate if incorporating the knowledge into electronic health records improves patient outcomes.

Citation: McCoy AB, Wright A, Krousel-Wood M, Thomas EJ, McCoy JA, Sittig DF. Validation of a crowdsourcing methodology for developing a knowledge base of related problem-medication pairs. Appl Clin Inf 2015; 6: 334–344

http://dx.doi.org/10.4338/ACI-2015-01-RA-0010

Keywords

Crowdsourcing - electronic health records - knowledge bases - problem-oriented medical records - computer-assisted drug therapy - validation studies

 

Conflict of Interest

The authors declare that they have no conflicts of interest in the research.

• References

• 1 Feblowitz JC, Wright A, Singh H, Samal L, Sittig DF. Summarization of clinical information: a conceptual model. J Biomed Inform 2011; 44 (04) 688-699.
• 2 McCoy AB, Wright A, Laxmisan A, Ottosen MJ, McCoy JA, Butten D, Sittig DF. Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications. J Am Med Inform Assoc JAMIA 2012; 19 (05) 713-718.
• 3 Bates DW, Cullen DJ, Laird N, Petersen LA, Small SD, Servi D, Laffel G, Sweitzer BJ, Shea BF, Hallisey R. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA J Am Med Assoc 1995; 274 (01) 29-34.
• 4 Blumenthal D, Tavenner M. The “Meaningful Use” Regulation for Electronic Health Records. N Engl J Med 2010; 363 (06) 501-504.
• 5 Laxmisan A, McCoy AB, Wright A, Sittig DF. Clinical Summarization Capabilities of Commercially-available and Internally-developed Electronic Health Records. Appl Clin Inform 2012; 3 (01) 80-93.
• 6 Health IT and Patient Safety: Building Safer Systems for Better Care –Institute of Medicine. Available from: http://iom.edu/Reports/2011/Health-IT-and-Patient-Safety-Building-Safer-Systems-for-Better-Care.aspx
• 7 Han YY, Carcillo JA, Venkataraman ST, Clark RSB, Watson RS, Nguyen TC, Bayir H, Orr RA. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512.
• 8 Horsky J, Kuperman GJ, Patel VL. Comprehensive analysis of a medication dosing error related to CPOE. J Am Med Inform Assoc JAMIA 2005; 12 (04) 377-382.
• 9 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL. Role of computerized physician order entry systems in facilitating medication errors. JAMA J Am Med Assoc 2005; 293 (10) 1197-1203.
• 10 McCoy AB, Waitman LR, Lewis JB, Wright JA, Choma DP, Miller RA, Peterson JF. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc JAMIA 2012; 19 (03) 346-352.
• 11 Ash JS, Berg M, Coiera E. Some Unintended Consequences of Information Technology in Health Care: The Nature of Patient Care Information System-related Errors. J Am Med Inform Assoc 2004; 11 (02) 104-112.
• 12 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc JAMIA 2007; 14 (04) 415-423.
• 13 Sittig DF, Singh H. Legal, ethical, and financial dilemmas in electronic health record adoption and use. Pediatrics 2011; 127 (04) e1042-e1047.
• 14 Sittig DF, Teich JM, Osheroff JA, Singh H. Improving clinical quality indicators through electronic health records: it takes more than just a reminder. Pediatrics 2009; 124 (01) 375-377.
• 15 Sittig DF, Singh H. Rights and responsibilities of users of electronic health records. CMAJ Can Med Assoc J J Assoc Medicale Can. 2012; 184 (13) 1479-1483.
• 16 Carter JS, Brown SH, Erlbaum MS, Gregg W, Elkin PL, Speroff T, Tuttle MS. Initializing the VA medication reference terminology using UMLS metathesaurus co-occurrences. Proc AMIA Symp 2002; 116-120.
• 17 Elkin PL, Carter JS, Nabar M, Tuttle M, Lincoln M, Brown SH. Drug knowledge expressed as computable semantic triples. Stud Health Technol Inform 2011; 166: 38-47.
• 18 Wright A, Chen ES, Maloney FL. An automated technique for identifying associations between medications, laboratory results and problems. J Biomed Inform 2010; 43 (06) 891-901.
• 19 Brown SH, Miller RA, Camp HN, Guise DA, Walker HK. Empirical derivation of an electronic clinically useful problem statement system. Ann Intern Med 1999; 131 (02) 117-126.
• 20 Zeng Q, Cimino JJ, Zou KH. Providing concept-oriented views for clinical data using a knowledge-based system: an evaluation. J Am Med Inform Assoc JAMIA 2002; 9 (03) 294-305.
• 21 Chen ES, Hripcsak G, Xu H, Markatou M, Friedman C. Automated acquisition of disease drug knowledge from biomedical and clinical documents: an initial study. J Am Med Inform Assoc JAMIA 2008; 15 (01) 87-98.
• 22 Kilicoglu H, Fiszman M, Rodriguez A, Shin D, Ripple A, Rindflesch TC. Semantic MEDLINE: A Web Application for Managing the Results of PubMed Searches. 2008: 69-76.
• 23 Duke JD, Friedlin J. ADESSA: A Real-Time Decision Support Service for Delivery of Semantically Coded Adverse Drug Event Data. AMIA Annu Symp Proc AMIA Symp AMIA Symp 2010; 2010: 177-181.
• 24 Wu Y, Wright A, Xu H, McCoy AB, Sittig DF. Development of a Unified Computable Problem-Medication Knowledge Base. AMIA Annu Symp Proc AMIA Symp AMIA Symp 2014; 2014.
• 25 Berk RA. An introduction to ensemble methods for data analysis. Sociol Methods Res 2006; 34 (03) 263-295.
• 26 Tapscott D. Wikinomics : how mass collaboration changes everything. New York: Portfolio; 2006
• 27 Howe J. The rise of crowdsourcing. Wired Mag 2006; 14 (06) 1-4.
• 28 Giles J. Internet encyclopaedias go head to head. Nature 2005; 438 7070 900-901.
• 29 Ekins S, Williams AJ. Reaching out to collaborators: crowdsourcing for pharmaceutical research. Pharm Res 2010; 27 (03) 393-395.
• 30 Hughes S, Cohen D. Can Online Consumers Contribute to Drug Knowledge? A Mixed-Methods Comparison of Consumer-Generated and Professionally Controlled Psychotropic Medication Information on the Internet. J Med Internet Res. 2011 13. 03
• 31 Brownstein CA, Brownstein JS, Williams DS, 3rd, Wicks P, Heywood JA. The power of social networking in medicine. Nat Biotechnol 2009; 27 (10) 888-890.
• 32 Parry DT, Tsung-Chun Tsai. Crowdsourcing techniques to create a fuzzy subset of SNOMED CT for semantic tagging of medical documents. 2010 IEEE International Conference on Fuzzy Systems (FUZZ). IEEE. 2010: 1-8.
• 33 Wagholikar KB, MacLaughlin KL, Kastner TM, Casey PM, Henry M, Greenes RA, Liu H, Chaudhry R. Formative evaluation of the accuracy of a clinical decision support system for cervical cancer screening. J Am Med Inform Assoc. 2013 Apr 5; amiajnl –2013-001613.
• 34 Good BM, Su AI. Crowdsourcing for. Bioinformatics 2013; btt333.
• 35 Sweidan M, Williamson M, Reeve JF, Harvey K, O’Neill JA, Schattner P, Snowdon T. Evaluation of features to support safety and quality in general practice clinical software. BMC Med Inform Decis Mak 2011; 11 (01) 27.
• 36 Hersh W. Evaluation of biomedical text-mining systems: lessons learned from information retrieval. Brief Bioinform 2005; 6 (04) 344-356.