Subscribe to RSS
DOI: 10.1055/a-1745-1348
External Validation of Postpartum Hemorrhage Prediction Models Using Electronic Health Record Data
Funding This study was funded by a University of Michigan internal grant: University of Michigan Precision Health Investigator Award 2019. K.K.V. was supported by the Care Innovation and Community Improvement Program at The Ohio State University. J.E.J. was supported by the NIDDK Symptoms of Lower Urinary Dysfunction Network at Duke University. These funding sources had no role in study design, data collection, analysis and interpretation, in writing of the report, and in the decision to submit the article for publication.![](https://www.thieme-connect.de/media/ajp/202405/lookinside/thumbnails/10-1055-a-1745-1348_210529-1.jpg)
Abstract
Objective A recent study leveraging machine learning methods found that postpartum hemorrhage (PPH) can be predicted accurately at the time of labor admission in the U.S. Consortium for Safe Labor (CSL) dataset, with a C-statistic as high as 0.93. These CSL models were developed in older data (2002–2008) and used an estimated blood loss (EBL) of ≥1,000 mL to define PPH. We sought to externally validate these models using a more recent cohort of births where blood loss was measured using quantitative blood loss (QBL) methods.
Study Design Using data from 5,261 deliveries between February 1, 2019 and May 11, 2020 at a single tertiary hospital, we mapped our electronic health record (EHR) data to the 55 predictors described in previously published CSL models. PPH was defined as QBL ≥1,000 mL within 24 hours after delivery. Model discrimination and calibration of the four CSL models were measured using our cohort. In a secondary analysis, we fit new models in our study cohort using the same predictors and algorithms as the original CSL models.
Results The original study cohort had a substantially lower rate of PPH, 4.8% (7,279/228,438) versus 25% (1,321/5,261), possibly due to differences in measurement. The CSL models had lower discrimination in our study cohort, with a C-statistic as high as 0.57 (logistic regression). Models refit in our study cohort achieved better discrimination, with a C-statistic as high as 0.64 (random forest). Calibration improved in the refit models as compared with the original models.
Conclusion The CSL models' accuracy was lower in a contemporary EHR where PPH is assessed using QBL. As institutions continue to adopt QBL methods, further data are needed to understand the differences between EBL and QBL to enable accurate prediction of PPH.
Key Points
-
Machine learning methods may help predict PPH.
-
EBL models do not generalize when QBL is used.
-
Blood loss estimation alters model accuracy.
Keywords
electronic health record - external validation - machine learning - postpartum hemorrhage - prediction model* These authors contributed equally.
Publication History
Received: 14 July 2021
Accepted: 17 January 2022
Accepted Manuscript online:
19 January 2022
Article published online:
02 March 2022
© 2022. Thieme. All rights reserved.
Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA
-
References
- 1 Reale SC, Easter SR, Xu X, Bateman BT, Farber MK. Trends in postpartum hemorrhage in the United States From 2010 to 2014. Anesth Analg 2020; 130 (05) e119-e122
- 2 Main EK, Goffman D, Scavone BM. et al. National partnership for maternal safety consensus bundle on obstetric hemorrhage. J Midwifery Womens Health 2015; 60 (04) 458-464
- 3 Bateman BT, Berman MF, Riley LE, Leffert LR. The epidemiology of postpartum hemorrhage in a large, nationwide sample of deliveries. Anesth Analg 2010; 110 (05) 1368-1373
- 4 Klumpner TT, Kountanis JA, Meyer SR. et al. Use of a novel electronic maternal surveillance system and the maternal early warning criteria to detect severe postpartum hemorrhage. Anesth Analg 2020; 131 (03) 857-865
- 5 Klumpner TT, Kountanis JA, Langen ES, Smith RD, Tremper KK. Use of a novel electronic maternal surveillance system to generate automated alerts on the labor and delivery unit. BMC Anesthesiol 2018; 18 (01) 78
- 6 Venkatesh KK, Strauss RA, Grotegut CA. et al. Machine learning and statistical models to predict postpartum hemorrhage. Obstet Gynecol 2020; 135 (04) 935-944
- 7 Bell SF, Watkins A, John M. et al. Incidence of postpartum haemorrhage defined by quantitative blood loss measurement: a national cohort. BMC Pregnancy Childbirth 2020; 20 (01) 271
- 8 Quantitative Blood Loss in Obstetric Hemorrhage. ACOG COMMITTEE OPINION SUMMARY, Number 794. Obstet Gynecol 2019; 134 (06) 1368-1369
- 9 Moreno-Torres JG, Raeder T, Alaiz-Rodríguez R, Chawla NV, Herrera F. A unifying view on dataset shift in classification. Pattern Recognition 2012; 45 (01) 521-530
- 10 Subbaswamy A, Saria S. From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics 2020; 21 (02) 345-352
- 11 Debray TPA, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KGM. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68 (03) 279-289
- 12 Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan. Accessed March 18, 2021 at: https://www.fda.gov/media/145022/download
- 13 reVITALize: Obstetrics Data Definitions. Accessed June 1, 2021 at: https://www.acog.org/practice-management/health-it-and-clinical-informatics/revitalize-obstetrics-data-definitions
- 14 Zhang J, Troendle J, Reddy UM. et al; Consortium on Safe Labor. Contemporary cesarean delivery practice in the United States. Am J Obstet Gynecol 2010; 203 (04) 326.e1-326.e10
- 15 Pencina MJ, D'Agostino Sr RB. Evaluating discrimination of risk prediction models: the C Statistic. JAMA 2015; 314 (10) 1063-1064
- 16 Breiman L.. Bagging predictors. Machine Learning 1996; 24 (02) 123-140
- 17 R Core Team. R: A Language and Environment for Statistical Computing. 2021 https://www.R-project.org
- 18 R Interface for the H2O Scalable Machine Learning Platform. Accessed September 28, 2020 at: https://docs.h2o.ai/h2o/latest-stable/h2o-r/docs/index.html
- 19 Singh K. ML4LHS/Runway. Machine Learning for Learning Health Systems Lab; 2020. Accessed November 19, 2020 at: https://github.com/ML4LHS/runway
- 20 Robin X, Turck N, Hainard A. et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 2011; 12 (01) 77
- 21 Meyer SR. Pph_model_validation. Github. Accessed November 21, 2021 at: https://github.com/seanrmeyer/pph_model_validation
- 22 Tangri N, Grams ME, Levey AS. et al; CKD Prognosis Consortium. Multinational assessment of accuracy of equations for predicting risk of kidney failure: a meta-analysis. JAMA 2016; 315 (02) 164-174
- 23 Steyerberg E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Second. Heidelberg, Germany: Springer-Verlag; 2019
- 24 Moons KGM, Altman DG, Reitsma JB. et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015; 162 (01) W1-73
- 25 Escobar GJ, Soltesz L, Schuler A, Niki H, Malenica I, Lee C. Prediction of obstetrical and fetal complications using automated electronic health record data. Am J Obstet Gynecol 2021; 224 (02) 137-147.e7