Predicting hospitalization following psychiatric crisis care using machine learning

Abstract

Background Accurate prediction models for whether patients on the verge of a psychiatric criseis need hospitalization are lacking and machine learning methods may help improve the accuracy of psychiatric hospitalization prediction models. In this paper we evaluate the accuracy of ten machine learning algorithms, including the generalized linear model (GLM/logistic regression) to predict psychiatric hospitalization in the first 12 months after a psychiatric crisis care contact. We also evaluate an ensemble model to optimize the accuracy and we explore individual predictors of hospitalization. Methods Data from 2084 patients included in the longitudinal Amsterdam Study of Acute Psychiatry with at least one reported psychiatric crisis care contact were included. Target variable for the prediction models was whether the patient was hospitalized in the 12 months following inclusion. The predictive power of 39 variables related to patients’ socio-demographics, clinical characteristics and previous mental health care contacts was evaluated. The accuracy and area under the receiver operating characteristic curve (AUC) of the machine learning algorithms were compared and we also estimated the relative importance of each predictor variable. The best and least performing algorithms were compared with GLM/logistic regression using net reclassification improvement analysis and the five best performing algorithms were combined in an ensemble model using stacking. Results All models performed above chance level. We found Gradient Boosting to be the best performing algorithm (AUC = 0.774) and K-Nearest Neighbors to be the least performing (AUC = 0.702). The performance of GLM/logistic regression (AUC = 0.76) was slightly above average among the tested algorithms. In a Net Reclassification Improvement analysis Gradient Boosting outperformed GLM/logistic regression by 2.9% and K-Nearest Neighbors by 11.3%. GLM/logistic regression outperformed K-Nearest Neighbors by 8.7%. Nine of the top-10 most important predictor variables were related to previous mental health care use. Conclusions Gradient Boosting led to the highest predictive accuracy and AUC while GLM/logistic regression performed average among the tested algorithms. Although statistically significant, the magnitude of the differences between the machine learning algorithms was in most cases modest. The results show that a predictive accuracy similar to the best performing model can be achieved when combining multiple algorithms in an ensemble model.

Predicting Hospitalization Risk of Suspected COVID-19 Patients Using a Machine Learning Methodology

“…Our proposed model performed an accuracy of 90.4%, an AUC of 0.959, a PPV of 0.933 and a NPV of 0.929 including 35 features. This result is better than the one found by [4], that found an AUC=0.774 including 39 features, and the one found by [5], that found an AUC=0.7. Our result is also better than the one found by [7], that found an AUC=0.813 in the validation cohort using 48 features.…”

“…Previous studies have shown promising results when applying machine learning algorithms to predict hospitalization [4], [5]. Blankers, van der Post and Dekker [4] found Gradient Boosting to be the best performing algorithm (Area Under the Curve (AUC)=0.774) including 39 variables related to patients' socio-demographics, clinical characteristics, and previous mental health care.…”

“…Previous studies have shown promising results when applying machine learning algorithms to predict hospitalization [4], [5]. Blankers, van der Post and Dekker [4] found Gradient Boosting to be the best performing algorithm (Area Under the Curve (AUC)=0.774) including 39 variables related to patients' socio-demographics, clinical characteristics, and previous mental health care. Wong et al [5] found Generalized Additive Models (GAM) to be the best model (AUC = 0.70, 95% confidence interval = 0.62−0.79) for hospitalization prediction, including nutritional and inflammation indicators, demographics, comorbidities, and slopes all continuous variables over 6 months as model predictors.…”

Developing a Machine Learning Risk-adjustment Method for Hospitalizations and Emergency Department Visits of Nursing Home Residents With Dementia

“…Our XGBoost models only achieved a minimum improvement compared with logistic regression in predicting hospitalization and ED visit, consistent with previous studies. 9,14,40,41 Interestingly, our models performed better than models used by Care Compare based on various metrics of performance. 5 There are multiple reasons why this may have occurred.…”

“…Our model performances exceed many existing predictive models, which reported AUC of around 0.7. 12,[39][40][41] One strength of our model is the inclusion of detailed resident-level predictors from MDS, 4 which gave us more power to predict hospital transfers. Our XGBoost models only achieved a minimum improvement compared with logistic regression in predicting hospitalization and ED visit, consistent with previous studies.…”