A scalable approach for developing clinical risk prediction applications in different hospitals

Sun, Hong; Depraetere, Kristof; Meesseman, Laurent; Roo, Jos De; Vanbiervliet, Martijn; Baerdemaeker, Jos De; Muys, Herman; Dossow, Vera von; Hulde, Nikolai; Szymanowsky, Ralph

doi:10.1016/j.jbi.2021.103783

Cited by 17 publications

(13 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We refer to these three hospitals as hospital M, hospital H, and hospital N, respectively. The calibration process that generates prediction models is described in our previous work [ 9 ]. Using the calibration tool, models were trained independently on data from each hospital and deployed in the prediction service of the same hospital.…”

Section: Methodsmentioning

confidence: 99%

“…In order to cope with the situation where the model was requested to make predictions when less information was available, we applied data augmentation in training sample preparation: we generated partial records in combination with the complete records to enhance the robustness of the clinical risk prediction model. More details can be found in our previous work [ 9 ]. The generated models were first examined with a model checking process, where a set of minimum requirements were assessed as unit tests.…”

Section: Methodsmentioning

confidence: 99%

“…Risk prediction models evaluated in this paper were generated by our calibration tool in the three aforementioned German hospitals. The details of feature engineering and model training were presented in a former publication [ 9 ]; examples of model input features are provided in Table S5 in Multimedia Appendix 1 . Preliminary cross-hospital evaluation was performed with retrospective data in our previous study and performance degradation was observed [ 9 ].…”

Section: Methodsmentioning

confidence: 99%

“…In our previous publication [ 9 ], we discussed the scalability issue in clinical risk prediction model development; we also presented a scalable approach for prediction model development that is applied to delirium, sepsis, and acute kidney injury (AKI) covering four different hospitals. However, these prediction models were only evaluated on retrospective data.…”

Section: Introductionmentioning

confidence: 99%

“…However, very few prediction models have been evaluated in a live clinical workflow. For example, several delirium prediction models that have been reported in recent years [ 9 , 12 , 13 ] have all been evaluated on retrospective data. Jauk et al [ 14 ] claimed their findings to be the only delirium prediction model that has been evaluated in a live clinical workflow.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Machine Learning–Based Prediction Models for Different Clinical Risks in Different Hospitals: Evaluation of Live Performance

Sun¹,

Depraetere²,

Meesseman³

et al. 2022

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

Background Machine learning algorithms are currently used in a wide array of clinical domains to produce models that can predict clinical risk events. Most models are developed and evaluated with retrospective data, very few are evaluated in a clinical workflow, and even fewer report performances in different hospitals. In this study, we provide detailed evaluations of clinical risk prediction models in live clinical workflows for three different use cases in three different hospitals. Objective The main objective of this study was to evaluate clinical risk prediction models in live clinical workflows and compare their performance in these setting with their performance when using retrospective data. We also aimed at generalizing the results by applying our investigation to three different use cases in three different hospitals. Methods We trained clinical risk prediction models for three use cases (ie, delirium, sepsis, and acute kidney injury) in three different hospitals with retrospective data. We used machine learning and, specifically, deep learning to train models that were based on the Transformer model. The models were trained using a calibration tool that is common for all hospitals and use cases. The models had a common design but were calibrated using each hospital’s specific data. The models were deployed in these three hospitals and used in daily clinical practice. The predictions made by these models were logged and correlated with the diagnosis at discharge. We compared their performance with evaluations on retrospective data and conducted cross-hospital evaluations. Results The performance of the prediction models with data from live clinical workflows was similar to the performance with retrospective data. The average value of the area under the receiver operating characteristic curve (AUROC) decreased slightly by 0.6 percentage points (from 94.8% to 94.2% at discharge). The cross-hospital evaluations exhibited severely reduced performance: the average AUROC decreased by 8 percentage points (from 94.2% to 86.3% at discharge), which indicates the importance of model calibration with data from the deployment hospital. Conclusions Calibrating the prediction model with data from different deployment hospitals led to good performance in live settings. The performance degradation in the cross-hospital evaluation identified limitations in developing a generic model for different hospitals. Designing a generic process for model development to generate specialized prediction models for each hospital guarantees model performance in different hospitals.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Machine Learning–Based Prediction Models for Different Clinical Risks in Different Hospitals: Evaluation of Live Performance

Sun¹,

Depraetere²,

Meesseman³

et al. 2022

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

show abstract

せん妄の予測に対する機械学習の利用状況：スコーピングレビュー(The use of machine learning for the prediction of delirium: a scoping review)

守克

和也

匡

et al. 2022

Nihon Kyukyu Igakukai Zasshi: Japanese Journal of Japanese Asso

View full text Add to dashboard Cite

要旨【目的】せん妄の予測に対して機械学習を利用した研究の動向を概観し，今後の課題を明らかにすることを目的とした。【方法】2021年4月までに発行された論文を対象として，PubMed，CINAHL，医中誌Webの各データベースで検索し，適格基準を満たした論文をレビューの対象とした。【結果】適格基準を満たした論文は23編であった。対象論文は多施設研究が多かったものの，集中治療患者を対象とした研究は単施設研究に限られていた。せん妄の予測には，多くの論文が100以上の変数を使用していた。せん妄のラベルにはICDを使用した論文が多かった。【結語】集中治療患者を対象として，予測に使用する変数を減らし，CAM–ICUやICDSCをせん妄のラベルとする多施設研究を実施することで，せん妄の予測における機械学習の利用可能性を模索する必要がある。

show abstract

Machine Learning–Based Prediction Models for Delirium: A Systematic Review and Meta-Analysis

Xie¹,

Wang²,

Pei³

et al. 2022

Journal of the American Medical Directors Association

View full text Add to dashboard Cite

A scalable approach for developing clinical risk prediction applications in different hospitals

Cited by 17 publications

References 32 publications

Machine Learning–Based Prediction Models for Different Clinical Risks in Different Hospitals: Evaluation of Live Performance

Machine Learning–Based Prediction Models for Different Clinical Risks in Different Hospitals: Evaluation of Live Performance

せん妄の予測に対する機械学習の利用状況：スコーピングレビュー(The use of machine learning for the prediction of delirium: a scoping review)

Machine Learning–Based Prediction Models for Delirium: A Systematic Review and Meta-Analysis

Contact Info

Product

Resources

About