External Validation of Postpartum Hemorrhage Prediction Models Using Electronic Health Record Data

Abstract

Objective: A recent study leveraging machine learning methods found that postpartum hemorrhage (PPH) can be predicted accurately at the time of labor admission in the U.S. Consortium for Safe Labor (CSL) dataset, with a C-statistic as high as 0.93. These CSL models were developed in older data (2002-2008) and used an estimated blood loss (EBL) of ≥1000 mL to define PPH. We sought to externally validate these models using a more recent cohort of births where blood loss was measured using quantitative blood loss (QBL) methods. Study Design: Using data from 5,261 deliveries between February 1, 2019 to May 11, 2020 at a single tertiary hospital, we mapped our electronic health record (EHR) data to the 55 predictors described in previously published CSL models. PPH was defined as QBL ≥1000 mL within 24 hours after delivery. Model discrimination and calibration of the four CSL models were measured using our cohort. In a secondary analysis, we fit new models in our study cohort using the same predictors and algorithms as the original CSL models. Results: The original study cohort had a substantially lower rate of PPH, 4.8% (7,279/228,438) vs. 25% (1,321/5,261), possibly due to differences in measurement. The CSL models had lower discrimination in our study cohort, with a C-statistic as high as 0.57 (logistic regression). Models refit in our study cohort achieved better discrimination, with a C-statistic as high as 0.64 (random forrest). Calibration improved in the refit models as compared to the original models. Conclusion: The CSL models’ accuracy was lower in a contemporary EHR where PPH is assessed using QBL. As institutions continue to adopt QBL methods, further data are needed to understand the differences between EBL and QBL to enable accurate prediction of PPH.

Machine Learning Models for Prediction of Maternal Hemorrhage and Transfusion: Model Development Study

“…A previously published risk assessment for PPH using the CSL data set demonstrated exceptional model performance, but model performance was drastically lower in an external validation cohort [ 21 , 22 ]. This study augments the findings of these prior studies via incorporation of antepartum and intrapartum risk factors.…”