The Potential Cost-Effectiveness of a Machine Learning Tool That Can Prevent Untimely Intensive Care Unit Discharge

Vos, Juliette de; Visser, Laurenske A.; Beer, Aletta A. de; Fornasa, Mattia; Thoral, Patrick; Elbers, Paul; Cinà, Giovanni

doi:10.1016/j.jval.2021.06.018

Cited by 16 publications

(10 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This survey study is part of preimplementation research for Pacmed Critical [24]. Pacmed Critical is a machine learning-based AI-CDS tool that predicts a patient's combined readmission and mortality risk within 7 days of ICU discharge to support physicians in their decisions to discharge patients to lower care wards [25,26]. The Pacmed Critical software is intended for use as a complementary tool by qualified ICU medical professionals and will be accessed on hospital premises; it will not be used on mobile devices.…”

Section: Introductionmentioning

confidence: 99%

Intensive Care Unit Physicians’ Perspectives on Artificial Intelligence–Based Clinical Decision Support Tools: Preimplementation Survey Study

Meijden¹,

Hond²,

Thoral³

et al. 2023

JMIR Hum Factors

Self Cite

View full text Add to dashboard Cite

Background Artificial intelligence–based clinical decision support (AI-CDS) tools have great potential to benefit intensive care unit (ICU) patients and physicians. There is a gap between the development and implementation of these tools. Objective We aimed to investigate physicians’ perspectives and their current decision-making behavior before implementing a discharge AI-CDS tool for predicting readmission and mortality risk after ICU discharge. Methods We conducted a survey of physicians involved in decision-making on discharge of patients at two Dutch academic ICUs between July and November 2021. Questions were divided into four domains: (1) physicians’ current decision-making behavior with respect to discharging ICU patients, (2) perspectives on the use of AI-CDS tools in general, (3) willingness to incorporate a discharge AI-CDS tool into daily clinical practice, and (4) preferences for using a discharge AI-CDS tool in daily workflows. Results Most of the 64 respondents (of 93 contacted, 69%) were familiar with AI (62/64, 97%) and had positive expectations of AI, with 55 of 64 (86%) believing that AI could support them in their work as a physician. The respondents disagreed on whether the decision to discharge a patient was complex (23/64, 36% agreed and 22/64, 34% disagreed); nonetheless, most (59/64, 92%) agreed that a discharge AI-CDS tool could be of value. Significant differences were observed between physicians from the 2 academic sites, which may be related to different levels of involvement in the development of the discharge AI-CDS tool. Conclusions ICU physicians showed a favorable attitude toward the integration of AI-CDS tools into the ICU setting in general, and in particular toward a tool to predict a patient’s risk of readmission and mortality within 7 days after discharge. The findings of this questionnaire will be used to improve the implementation process and training of end users.

show abstract

Section: Introductionmentioning

confidence: 99%

Intensive Care Unit Physicians’ Perspectives on Artificial Intelligence–Based Clinical Decision Support Tools: Preimplementation Survey Study

Meijden¹,

Hond²,

Thoral³

et al. 2023

JMIR Hum Factors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Poor generalizability of ML models from one local setting to another limits the scalability of these techniques ( 21 ). The potential of Pacmed Critical ( 33 ) may not come to fruition by nontransportable and highly tailored solutions that are labor-intensive to develop and maintain. Future research should analyze multisite datasets to explore heterogeneity in predictive relations as threats to developing generalizable models ( 34 ).…”

Section: Discussionmentioning

confidence: 99%

Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model

Hond

Kant

Fornasa³

et al. 2022

Critical Care Medicine

Self Cite

View full text Add to dashboard Cite

OBJECTIVES: Many machine learning (ML) models have been developed for application in the ICU, but few models have been subjected to external validation. The performance of these models in new settings therefore remains unknown. The objective of this study was to assess the performance of an existing decision support tool based on a ML model predicting readmission or death within 7 days after ICU discharge before, during, and after retraining and recalibration. DESIGN:A gradient boosted ML model was developed and validated on electronic health record data from 2004 to 2021. We performed an independent validation of this model on electronic health record data from 2011 to 2019 from a different tertiary care center. SETTING: Two ICUs in tertiary care centers in The Netherlands. PATIENTS:Adult patients who were admitted to the ICU and stayed for longer than 12 hours. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS:We assessed discrimination by area under the receiver operating characteristic curve (AUC) and calibration (slope and intercept). We retrained and recalibrated the original model and assessed performance via a temporal validation design. The final retrained model was cross-validated on all data from the new site. Readmission or death within 7 days after ICU discharge occurred in 577 of 10,052 ICU admissions (5.7%) at the new site. External validation revealed moderate discrimination with an AUC of 0.72 (95% CI 0.67-0.76). Retrained models showed improved discrimination with AUC 0.79 (95% CI 0.75-0.82) for the final validation model. Calibration was poor initially and good after recalibration via isotonic regression. CONCLUSIONS:In this era of expanding availability of ML models, external validation and retraining are key steps to consider before applying ML models to new settings. Clinicians and decision-makers should take this into account when considering applying new ML models to their local settings.

show abstract

“…We based our procedure on that of Tonekaboni et al ,21 who introduced clinicians to a hypothetical scenario of an implemented AI model as the starting point of an interview. We described three types of AI models (decision support for diagnoses, optimising resources and personalised medicine) that are implemented in clinical practice within the fields of radiology, intensive care unit and internal medicine 22–26. These model types were translated to common scenarios in obstetrics where such AI models could be used in practice.…”

Section: Methodsmentioning

confidence: 99%

What is the future of artificial intelligence in obstetrics? A qualitative study among healthcare professionals

Fischer,

Rietveld,

Teunissen

et al. 2023

BMJ Open

View full text Add to dashboard Cite

ObjectiveThis work explores the perceptions of obstetrical clinicians about artificial intelligence (AI) in order to bridge the gap in uptake of AI between research and medical practice. Identifying potential areas where AI can contribute to clinical practice, enables AI research to align with the needs of clinicians and ultimately patients.DesignQualitative interview study.SettingA national study conducted in the Netherlands between November 2022 and February 2023.ParticipantsDutch clinicians working in obstetrics with varying relevant work experience, gender and age.AnalysisThematic analysis of qualitative interview transcripts.ResultsThirteen gynaecologists were interviewed about hypothetical scenarios of an implemented AI model. Thematic analysis identified two major themes: perceived usefulness and trust. Usefulness involved AI extending human brain capacity in complex pattern recognition and information processing, reducing contextual influence and saving time. Trust required validation, explainability and successful personal experience. This result shows two paradoxes: first, AI is expected to provide added value by surpassing human capabilities, yet also a need to understand the parameters and their influence on predictions for trust and adoption was expressed. Second, participants recognised the value of incorporating numerous parameters into a model, but they also believed that certain contextual factors should only be considered by humans, as it would be undesirable for AI models to use that information.ConclusionsObstetricians’ opinions on the potential value of AI highlight the need for clinician-AI researcher collaboration. Trust can be built through conventional means like randomised controlled trials and guidelines. Holistic impact metrics, such as changes in workflow, not just clinical outcomes, should guide AI model development. Further research is needed for evaluating evolving AI systems beyond traditional validation methods.

show abstract

The Potential Cost-Effectiveness of a Machine Learning Tool That Can Prevent Untimely Intensive Care Unit Discharge

Cited by 16 publications

References 36 publications

Intensive Care Unit Physicians’ Perspectives on Artificial Intelligence–Based Clinical Decision Support Tools: Preimplementation Survey Study

Intensive Care Unit Physicians’ Perspectives on Artificial Intelligence–Based Clinical Decision Support Tools: Preimplementation Survey Study

Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model

What is the future of artificial intelligence in obstetrics? A qualitative study among healthcare professionals

Contact Info

Product

Resources

About