Methods Inf Med 2019; 58(01): 031-041
DOI: 10.1055/s-0039-1677692
Original Article
Georg Thieme Verlag KG Stuttgart · New York

Deep Learning versus Conventional Machine Learning for Detection of Healthcare-Associated Infections in French Clinical Narratives

Sara Rabhi
1   Telecom SudParis, Institut Mines-Telecom, Paris, Île-de-France, France
,
Jérémie Jakubowicz
1   Telecom SudParis, Institut Mines-Telecom, Paris, Île-de-France, France
,
Marie-Helene Metzger
2   INSERM U1018, Villejuif, France
3   Assistance Publique - Hôpitaux de Paris, Hôpital Antoine-Béclère, Clamart, France
4   Université Paris 13, UFR SMBH, Bobigny, France
› Author Affiliations
Funding This work was partly funded by the French National Research Agency, as part of its TECSAN program (ANR-08-TECS-001 and ANR-12-TECS-0006).
Further Information

Publication History

02 July 2018

04 December 2018

Publication Date:
15 March 2019 (online)

Abstract

Objective The objective of this article was to compare the performances of health care-associated infection (HAI) detection between deep learning and conventional machine learning (ML) methods in French medical reports.

Methods The corpus consisted in different types of medical reports (discharge summaries, surgery reports, consultation reports, etc.). A total of 1,531 medical text documents were extracted and deidentified in three French university hospitals. Each of them was labeled as presence (1) or absence (0) of HAI. We started by normalizing the records using a list of preprocessing techniques. We calculated an overall performance metric, the F1 Score, to compare a deep learning method (convolutional neural network [CNN]) with the most popular conventional ML models (Bernoulli and multi-naïve Bayes, k-nearest neighbors, logistic regression, random forests, extra-trees, gradient boosting, support vector machines). We applied the hyperparameter Bayesian optimization for each model based on its HAI identification performances. We included the set of text representation as an additional hyperparameter for each model, using four different text representations (bag of words, term frequency–inverse document frequency, word2vec, and Glove).

Results CNN outperforms all other conventional ML algorithms for HAI classification. The best F1 Score of 97.7% ± 3.6% and best area under the curve score of 99.8% ± 0.41% were achieved when CNN was directly applied to the processed clinical notes without a pretrained word2vec embedding. Through receiver operating characteristic curve analysis, we could achieve a good balance between false notifications (with a specificity equal to 0.937) and system detection capability (with a sensitivity equal to 0.962) using the Youden's index reference.

Conclusions The main drawback of CNNs is their opacity. To address this issue, we investigated CNN inner layers' activation values to visualize the most meaningful phrases in a document. This method could be used to build a phrase-based medical assistant algorithm to help the infection control practitioner to select relevant medical records. Our study demonstrated that deep learning approach outperforms other classification learning algorithms for automatically identifying HAIs in medical reports.

Supplementary Material

 
  • References

  • 1 Horan TC, Andrus M, Dudeck MA. CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting. Am J Infect Control 2008; 36 (05) 309-332
  • 2 Wilson J, Ramboer I, Suetens C. ; HELICS-SSI working group. Hospitals in Europe Link for Infection Control through Surveillance (HELICS). Inter-country comparison of rates of surgical site infection–opportunities and limitations. J Hosp Infect 2007; 65 (Suppl. 02) 165-170
  • 3 Suetens C, Morales I, Savey A. , et al. European surveillance of ICU-acquired infections (HELICS-ICU): methods and main results. J Hosp Infect 2007; 65 (Suppl. 02) 171-173
  • 4 Chalfine A, Cauet D, Lin WC. , et al. Highly sensitive and efficient computer-assisted system for routine surveillance for surgical site infection. Infect Control Hosp Epidemiol 2006; 27 (08) 794-801
  • 5 Gastmeier P, Bräuer H, Hauer T, Schumacher M, Daschner F, Rüden H. How many nosocomial infections are missed if identification is restricted to patients with either microbiology reports or antibiotic administration?. Infect Control Hosp Epidemiol 1999; 20 (02) 124-127
  • 6 Cadwallader HL, Toohey M, Linton S, Dyson A, Riley TV. A comparison of two methods for identifying surgical site infections following orthopaedic surgery. J Hosp Infect 2001; 48 (04) 261-266
  • 7 Bouzbid S, Gicquel Q, Gerbier S. , et al. Automated detection of nosocomial infections: evaluation of different strategies in an intensive care unit 2000-2006. J Hosp Infect 2011; 79 (01) 38-43
  • 8 Branch-Elliman W, Strymish J, Kudesia V, Rosen AK, Gupta K. Natural language processing for real-time catheter-associated urinary tract infection surveillance: results of a pilot implementation trial. Infect Control Hosp Epidemiol 2015; 36 (09) 1004-1010
  • 9 Proux D, Marchal P, Segond F. , et al. Natural language processing to detect risk patterns related to hospital acquired infections. In: International Workshop Biomedical Information Extraction: 2009; Borovets, Bulgaria: Association for Computational Linguistics; 2009:35–41
  • 10 Hagège C, Marchal P, Gicquel Q, Darmoni S, Pereira S, Metzger M. Linguistic and temporal processing for discovering hospital acquired infection from patient records. In: Knowledge Representation for Health-Care - ECAI 2010 Workshop KR4HC 2010, Lisbon, Portugal, August 17, 2010, Revised Selected Papers Edited by David Riaño AtT, Silvia Miksch, Mor Peleg, Vol. 6512. Springer; 2010: 70-84
  • 11 Gundlapalli AV, Divita G, Redd A. , et al. Detecting the presence of an indwelling urinary catheter and urinary symptoms in hospitalized patients using natural language processing. J Biomed Inform 2017; 71S: S39-S45
  • 12 Tvardik N, Kergourlay I, Bittar A, Segond F, Darmoni S, Metzger M-H. Accuracy of using natural language processing methods for identifying healthcare-associated infections. Int J Med Inform 2018; 117: 96-102
  • 13 Sips ME, Bonten MJM, van Mourik MSM. Automated surveillance of healthcare-associated infections: state of the art. Curr Opin Infect Dis 2017; 30 (04) 425-431
  • 14 Ehrentraut C, Ekholm M, Tanushi H, Tiedemann J, Dalianis H. Detecting hospital-acquired infections: a document classification approach using support vector machines and gradient tree boosting. Health Informatics J 2018; 24 (01) 24-42
  • 15 Sohn S, Larson DW, Habermann EB, Naessens JM, Alabbad JY, Liu H. Detection of clinically important colorectal surgical site infection using Bayesian network. J Surg Res 2017; 209: 168-173
  • 16 Shickel B, Tighe P, Bihorac A, Rashidi P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform 2018; 22 (05) 1589-1604
  • 17 Miotto R, Li L, Kidd BA, Dudley JT. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci Rep 2016; 6: 26094
  • 18 Miotto R, Wang F, Wang S, Jiang X, Dudley JT. Deep learning for healthcare: review, opportunities and challenges. Brief Bioinform 2018; 19 (06) 1236-1246
  • 19 Cocos A, Fiks AG, Masino AJ. Deep learning for pharmacovigilance: recurrent neural network architectures for labeling adverse drug reactions in Twitter posts. J Am Med Inform Assoc 2017; 24 (04) 813-821
  • 20 Habibi M, Weber L, Neves M, Wiegandt DL, Leser U. Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics 2017; 33 (14) i37-i48
  • 21 Jauregi Unanue I, Zare Borzeshi E, Piccardi M. Recurrent neural networks with specialized word embeddings for health-domain named-entity recognition. J Biomed Inform 2017; 76: 102-109
  • 22 Liu Z, Yang M, Wang X. , et al. Entity recognition from clinical texts via recurrent neural network. BMC Med Inform Decis Mak 2017; 17 (Suppl. 02) 67
  • 23 Luo Y. Recurrent neural networks for classifying relations in clinical notes. J Biomed Inform 2017; 72: 85-95
  • 24 Lyu C, Chen B, Ren Y, Ji D. Long short-term memory RNN for biomedical named entity recognition. BMC Bioinformatics 2017; 18 (01) 462
  • 25 Xie J, Liu X, Dajun Zeng D. Mining e-cigarette adverse events in social media using Bi-LSTM recurrent neural network with word embedding representation. J Am Med Inform Assoc 2018; 25 (01) 72-80
  • 26 Gehrmann S, Dernoncourt F, Li Y. , et al. Comparing rule-based and deep learning models for patient phenotyping. Comput Sci 2017; 15: 37
  • 27 Proux D, Hagège C, Gicquel Q. , et al. Architecture and systems for monitoring hospital acquired infections inside a hospital information workflow. In: 8th Conference on Recent Advances in Natural Language Processing. Hissar, Bulgaria; 2011:43–48
  • 28 RAISIN-Réseau-d'alerte-d'investigation-et-de-surveillance-des-infections-nosocomiales. Réseau ISO Raisin - Protocole de surveillance des infections du site opératoire - année 2011. RAISIN - INVS; 2011
  • 29 RAISIN-Réseau-d'alerte-d'investigation-et-de-surveillance-des-infections-nosocomiales. Réseau REA Raisin - Protocole de surveillance des infections nosocomiales en Réanimation Adulte - année 2011. RAISIN - INVS; 2011
  • 30 Khan A, Baharudin B, Lee L, Khan K. A review of machine learning algorithms for text-documents classification. J Adv Inform Tech 2010; 1 (01) 4-20
  • 31 Jivani A. A comparative study of stemming algorithms. Int J Comp Tech Appl 2011; 2 (06) 1930-1938
  • 32 Sparck Jones K. A statistical interpretation of term specificity and its application in retrieval. J Doc 1972; 28 (01) 11-21
  • 33 Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. 2013. Available at: https://arxiv.org/abs/1301.3781. Accessed January 19, 2019
  • 34 Rehurek R, Sojka P. Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on new challenges for NLP frameworks. Valletta, Malta: ELRA; 2010:45–50
  • 35 Pennington J, Socher R, Manning C. GloVe: Global Vectors for word representation. In: Conference on Empirical Methods on Natural Language Processing. Doha, Qatar; 2014:1532–1543
  • 36 Geurts P, Ernst D, Wehenkel L. Extremely randomized trees. Mach Learn 2006; 63: 3-42
  • 37 Friedman J. Greedy function approximation: a gradient boosting machine. Ann Stat 2001; 29 (05) 1189-1232
  • 38 Breiman L. Random forests. Mach Learn 2001; 45: 5-32
  • 39 McCallum A. K. N: a comparison of event models for Naive Bayes text classification. In: Proceedings of the AAAI-98 Workshop on Learning for Text Categorization; 1998:41–48
  • 40 Furey TS, Cristianini N, Duffy N, Bednarski DW, Schummer M, Haussler D. Support vector machine classification and validation of cancer tissue samples using microarray expression data. Bioinformatics 2000; 16 (10) 906-914
  • 41 Han E, Karypis G, Kumar V. Text categorization using weight-adjusted k-nearest neighbor classification. In: Proceedings PAKDD-01, 5th Pacific–Asia Conference on Knowledge Discovery and Data Mining, Vol. 2035, Springer ed.: Lecture Notes in Computer Science Series; 2001:53–65
  • 42 Kim Y. Convolutional neural networks for sentence classification. 2014. Available at: https://arxiv.org/abs/1408.5882. Accessed January 19, 2019
  • 43 Bergstra J, Komer B, Eliasmith C, Yamins D, Cox D. Hyperopt: a Python library for model selection and hyperparameter optimization. Comput Sci Discov 2015; 8 (01) 1-24
  • 44 Bergstra J, Bardenet R, Bengio Y, Kégl B. Algorithms for hyper-parameter optimization. In: Advances in Neural Information Processing Systems 24 (NIPS 2011); 2011
  • 45 Ruopp MD, Perkins NJ, Whitcomb BW, Schisterman EF. Youden Index and optimal cut-point estimated from observations affected by a lower limit of detection. Biom J 2008; 50 (03) 419-430
  • 46 Chinchor N, Sundheim B. MUC-5 evaluation metrics. In Proceedings of the Fifth Message Understanding Conference (MUC-5). Morgan Kaufmann, San Mateo, CA, 1993. Available at: http://www.aclweb.org/anthology/M93-1007 Accessed January 19, 2019
  • 47 Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput 1997; 9 (08) 1735-1780
  • 48 Cho K, van Merrienboer B, Gulcehre C. , et al. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Empirical Methods in Natural Language Processing. Doha, Qatar; 2014: 1724-1734
  • 49 Young T, Hazarika D, Poria S, Cambria E. Recent trends in deep learning based natural language processing; 2017. Available at: https://arxiv.org/abs/1708.02709 . Accessed December 28, 2018
  • 50 Yin W, Kann K, Yu M, Schütze H. Comparative study of CNN and RNN for Natural Language Processing; 2017. Available at: https://arxiv.org/abs/1702.01923 . Accessed December 28, 2018
  • 51 Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. In: Proceedings of ICLR-2015; 2015
  • 52 Yin W, Schütze H, Xiang B, Zhou B. ABCNN: attention-based convolutional neural network for modeling sentence pairs. Trans Assoc Comput Linguist 2016; 4: 259-272
  • 53 Mullenbach J, Wiegreffe S, Duke J, Sun J, Eisenstein J. Explainable prediction of medical codes from clinical text; 2018. Available at: https://arxiv.org/pdf/1802.05695.pdf . Accessed December 28, 2018
  • 54 Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E. Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; 2016
  • 55 Pham T, Tran T, Phung D, Venkatesh S. Predicting healthcare trajectories from medical records: a deep learning approach. J Biomed Inform 2017; 69: 218-229