Integrating human services and criminal justice data with claims data to predict risk of opioid overdose among Medicaid beneficiaries: A machine-learning approach

Abstract

Health system data incompletely capture the social risk factors for drug overdose. This study aimed to improve the accuracy of a machine-learning algorithm to predict opioid overdose risk by integrating human services and criminal justice data with health claims data to capture the social determinants of overdose risk. This prognostic study included Medicaid beneficiaries (n = 237,259) in Allegheny County, Pennsylvania enrolled between 2015 and 2018, randomly divided into training, testing, and validation samples. We measured 290 potential predictors (239 derived from Medicaid claims data) in 30-day periods, beginning with the first observed Medicaid enrollment date during the study period. Using a gradient boosting machine, we predicted a composite outcome (i.e., fatal or nonfatal opioid overdose constructed using medical examiner and claims data) in the subsequent month. We compared prediction performance between a Medicaid claims only model to one integrating human services and criminal justice data with Medicaid claims (i.e., integrated model) using several metrics (e.g., C-statistic, number needed to evaluate [NNE] to identify one overdose). Beneficiaries were stratified into risk-score decile subgroups. The samples (training = 79,087, testing = 79,086, validation = 79,086) had similar characteristics (age = 38±18 years, female = 56%, white = 48%, having at least one overdose = 1.7% during study period). Using the validation sample, the integrated model slightly improved on the Medicaid claims only model (C-statistic = 0.885; 95%CI = 0.877–0.892 vs. C-statistic = 0.871; 95%CI = 0.863–0.878), with small corresponding improvements in the NNE and positive predictive value. Nine of the top 30 most important predictors in the integrated model were human services and criminal justice variables. Using the integrated model, approximately 70% of individuals with overdoses were members of the top risk decile (overdose rates in the subsequent month = 47/10,000 beneficiaries). Few individuals in the bottom 9 deciles had overdose episodes (0-12/10,000). Machine-learning algorithms integrating claims and social service and criminal justice data modestly improved opioid overdose prediction among Medicaid beneficiaries for a large U.S. county heavily affected by the opioid crisis.

Developing and validating a machine-learning algorithm to predict opioid overdose in Medicaid beneficiaries in two US states: a prognostic modelling study

“…We applied several commonly used methods such as multivariate logistic regression, penalised regression, random forests, and gradient-boosting machine (GBM) to develop and test overdose prediction algorithms. Consistent with previous studies, 3 , 5 GBM yielded the best prediction results (C-statistic of 0·841 for GBM vs up to 0·820 for other methods; appendix p 11 ) with an ability to handle complex interactions between predictors and outcomes. The study’s objective was to externally validate the best-performing algorithm; thus, we focused on reporting the GBM model ( appendix pp 1 – 2 ).…”

“…This study expanded our previous work using machine-learning approaches to improve accuracy of predicting overdose in the subsequent 3 months in a large state Medicaid dataset and broaden applicability of these models across state Medicaid programmes. 5 Our best-performing GBM has several advantages, including handling missing data automatically, no additional feature selection process required prior to the GBM modelling, greater flexibility in hyper parameter tuning to include complex interactions between predictors and outcomes, and often providing better performance compared with other approaches. 3 , 5 We acknowledge, however, that the flexibility during model tuning can be time-consuming and computationally expensive.…”

“… 5 Our best-performing GBM has several advantages, including handling missing data automatically, no additional feature selection process required prior to the GBM modelling, greater flexibility in hyper parameter tuning to include complex interactions between predictors and outcomes, and often providing better performance compared with other approaches. 3 , 5 We acknowledge, however, that the flexibility during model tuning can be time-consuming and computationally expensive.…”

“…However, few studies have developed prediction algorithms to identify Medicaid beneficiaries at high risk of overdose. 5 – 7 Furthermore, little is known about whether prediction algorithms developed using earlier years from one US state will perform as well using more recent data or when applied to another state. 8…”

Accelerating Opioid Use Disorders Research by Integrating Multiple Data Modalities

“…This plethora of data can aid with in-depth phenotyping, including developing predictive models to identify at-risk individuals [21,33,34,58,[65][66][67][68][69][70][71]. The scalability of these big data efforts can allow for the identification of novel clusters and etiologies for OUD [68].…”