Can we reliably automate clinical prognostic modelling? A retrospective cohort study for ICU triage prediction of in-hospital mortality of COVID-19 patients in the Netherlands

Abstract

Background Building Machine Learning (ML) models in healthcare may suffer from time-consuming and potentially biased pre-selection of predictors by hand that can result in limited or trivial selection of suitable models. We aimed to assess the predictive performance of automating the process of building ML models (AutoML) in-hospital mortality prediction modelling of triage COVID-19 patients at ICU admission versus expert-based predictor pre-selection followed by logistic regression. Methods We conducted an observational study of all COVID-19 patients admitted to Dutch ICUs between February and July 2020. We included 2,690 COVID-19 patients from 70 ICUs participating in the Dutch National Intensive Care Evaluation (NICE) registry. The main outcome measure was in-hospital mortality. We asessed model performance (at admission and after 24 hours, respectively) of AutoML compared to the more traditional approach of predictor pre-selection and logistic regression. Findings : Predictive performance of the autoML models with variables available at admission shows fair discrimination (average AUROC = 0·75-0·76 (sdev = 0·03), PPV = 0·70-0·76 (sdev = 0·1) at cut-off = 0·3 (the observed mortality rate), and good calibration. This performance is on par with a logistic regression model with selection of patient variables by three experts (average AUROC = 0·78 (sdev = 0·03) and PPV = 0·79 (sdev = 0·2)). Extending the models with variables that are available at 24 hours after admission resulted in models with higher predictive performance (average AUROC = 0·77-0·79 (sdev = 0·03) and PPV = 0·79-0·80 (sdev = 0·10-0·17)). Conclusions AutoML delivers prediction models with fair discriminatory performance, and good calibration and accuracy, which is as good as regression models with expert-based predictor pre-selection. In the context of the restricted availability of data in an ICU quality registry, extending the models with variables that are available at 24 hours after admission showed small (but significantly) performance increase.

Development and evaluation of regression tree models for predicting in-hospital mortality of a national registry of COVID-19 patients over six pandemic surges

“… The 0 h and 24 h logistic regression (LR) models are reported in two other articles. The LR model at admission (0 h) is included in Supplementary Table 4 of [ 9 ] and the LR model at 24 h is included in Supplementary Table s 2 of [ 10 ]. The probabilities of the regression tree are in good agreement with the means and medians of the predictions of the LR model.…”

“…Other studies have also used machine learning algorithms for predicting ICU admission and mortality among COVID-19 patients. So did [ 16 ] compare the performance of 18 machine learning algorithms and [ 9 ] used automated machine learning with 20 algorithms for ICU triage prediction of in-hospital mortality of COVID-19 patients. Still, since our objective was to deliver simple interpretable models that could be easily used in practice, more advanced machine learning methods were not in scope for this study.…”

Use of machine learning for triage and transfer of ICU patients in the Covid-19 pandemic period: Scope Review

“…The study by Vagliano et al 15 , proposes automated machine learning design (AutoML), which includes automatic selection of ML models. Manual variable selection was discarded as restrictive, inefficient, and biased.…”

“…In particular, they demonstrated how the blood count collection through routine blood tests can be used for ML models, in situations of limited daily screening resources. The researchers intend to validate their models externally, with data from other hospitals and other periods of time, including different treatments and therapies applied to patients 15 .…”

Predicting Carpark Prices Indices in Hong Kong Using AutoML

“…Most AI and machine learning approaches need fine-tuning. Automated machine learning (AutoML) is a promising solution for building a deep learning system in the absence of human effort and has been applied in many different fields [146], such as finance [147] and ICU (intensive care unit) triage prediction [148]. The automated model selection method in AutoML includes feature engineering and neural architecture searching; AutoML streamlines the construction and application of machine learning models and significantly decreases the time, and improves the customized models' accuracy by reducing human errors [149].…”