Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): a retrospective deep learning study

Afshar, Majid; Sharma, Brihat; Dligach, Dmitriy; Oguss, Madeline; Brown, Randall; Chhabra, Neeraj; Thompson, Hale M.; Markossian, Talar; Joyce, Cara; Churpek, Matthew M; Karnik, Niranjan S.

doi:10.1016/s2589-7500(22)00041-3

Cited by 28 publications

(36 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We previously published a substance misuse screening tool using NLP and machine learning from the clinical notes, Substance Misuse Algorithm for Referral to Treatment using Artificial Intelligence (SMART-AI) [ 8 ]. SMART-AI was developed on hospitalized RUMC patients between October 1, 2017, and December 31, 2019, with temporal validation between January 1, 2020, and December 31, 2020.…”

Section: Methodsmentioning

confidence: 99%

“…The EHR system at RUMC is provided through Epic (Epic Systems Corporation). We designed an approach to collect notes from the first 24 hours of hospitalization from Epic, which is how SMART-AI was originally developed and validated [ 8 ]. With an average time of 1.6 days from a patient’s admission to receipt of a SUIT consultation during usual care, we anticipate this is sufficient time for the automated screener to operate and clinical interventions to occur after admission notes are collected.…”

Section: Methodsmentioning

confidence: 99%

“…SMART-AI will provide the final output classification for screen-positive cases for unhealthy alcohol, opioid, and/or nonopioid drug use. We will use a cutoff of 0.05 on the predicted probabilities for each substance use label because this provided the best test characteristics in the validation study [ 8 ]. The previously trained and validated CNN model for SMART-AI is available on Github [ 25 ], and more technical details about the model are detailed in the development and validation study [ 8 ].…”

Section: Methodsmentioning

confidence: 99%

“…Hospitals currently screening for unhealthy substance use need better approaches to identifying and treating patients, with less than a quarter of patients with a substance use disorder receiving treatment [ 7 ]. During the COVID-19 pandemic, screening efforts became even more challenging with changes in workflow and the reallocation of resources that further reduced manual screening rates [ 8 ]. Meanwhile, substance misuse ranks second among principal diagnoses for unplanned 7-day hospital readmission rates [ 7 , 9 ].…”

Section: Introductionmentioning

confidence: 99%

“…The computational methods of NLP derive semantic features from clinical notes, from which machine learning models can predict substance misuse. We previously published and made publicly available an NLP screening tool for different types of substance misuse [ 8 ]. During the validation of the algorithm, we achieved sensitivity and specificity greater than 85% using a convolutional neural network (CNN) from clinical notes with a false negative rate of less than 5% to screen for unhealthy alcohol use, unhealthy opioid use, and unhealthy nonopioid drug use (ie, cocaine and amphetamines).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

The Evaluation of a Clinical Decision Support Tool Using Natural Language Processing to Screen Hospitalized Adults for Unhealthy Substance Use: Protocol for a Quasi-Experimental Design

et al. 2022

Self Cite

View full text Add to dashboard Cite

Background Automated and data-driven methods for screening using natural language processing (NLP) and machine learning may replace resource-intensive manual approaches in the usual care of patients hospitalized with conditions related to unhealthy substance use. The rigorous evaluation of tools that use artificial intelligence (AI) is necessary to demonstrate effectiveness before system-wide implementation. An NLP tool to use routinely collected data in the electronic health record was previously validated for diagnostic accuracy in a retrospective study for screening unhealthy substance use. Our next step is a noninferiority design incorporated into a research protocol for clinical implementation with prospective evaluation of clinical effectiveness in a large health system. Objective This study aims to provide a study protocol to evaluate health outcomes and the costs and benefits of an AI-driven automated screener compared to manual human screening for unhealthy substance use. Methods A pre-post design is proposed to evaluate 12 months of manual screening followed by 12 months of automated screening across surgical and medical wards at a single medical center. The preintervention period consists of usual care with manual screening by nurses and social workers and referrals to a multidisciplinary Substance Use Intervention Team (SUIT). Facilitated by a NLP pipeline in the postintervention period, clinical notes from the first 24 hours of hospitalization will be processed and scored by a machine learning model, and the SUIT will be similarly alerted to patients who flagged positive for substance misuse. Flowsheets within the electronic health record have been updated to capture rates of interventions for the primary outcome (brief intervention/motivational interviewing, medication-assisted treatment, naloxone dispensing, and referral to outpatient care). Effectiveness in terms of patient outcomes will be determined by noninferior rates of interventions (primary outcome), as well as rates of readmission within 6 months, average time to consult, and discharge rates against medical advice (secondary outcomes) in the postintervention period by a SUIT compared to the preintervention period. A separate analysis will be performed to assess the costs and benefits to the health system by using automated screening. Changes from the pre- to postintervention period will be assessed in covariate-adjusted generalized linear mixed-effects models. Results The study will begin in September 2022. Monthly data monitoring and Data Safety Monitoring Board reporting are scheduled every 6 months throughout the study period. We anticipate reporting final results by June 2025. Conclusions The use of augmented intelligence for clinical decision support is growing with an increasing number of AI tools. We provide a research protocol for prospective evaluation of an automated NLP system for screening unhealthy substance use using a noninferiority design to demonstrate comprehensive screening that may be as effective as manual screening but less costly via automated solutions. Trial Registration ClinicalTrials.gov NCT03833804; https://clinicaltrials.gov/ct2/show/NCT03833804 International Registered Report Identifier (IRRID) DERR1-10.2196/42971

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

The Evaluation of a Clinical Decision Support Tool Using Natural Language Processing to Screen Hospitalized Adults for Unhealthy Substance Use: Protocol for a Quasi-Experimental Design

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

Towards Data-Driven Artificial Intelligence Models for Monitoring, Modelling and Predicting Illicit Substance Use

Mbunge,

Batani,

Chitungo

et al. 2024

Lecture Notes in Networks and Systems

View full text Add to dashboard Cite

A Reservoir Computing with Boosted Topology Model to Predict Encephalitis and Mortality for Patients with Severe Fever with Thrombocytopenia Syndrome: A Retrospective Multicenter Study

Zheng

Geng

et al. 2023

Infect Dis Ther

View full text Add to dashboard Cite

Introduction Severe fever with thrombocytopenia syndrome virus (SFTSV) is an emerging tick-borne virus associated with a high rate of mortality, as well as encephalitis. We aim to develop and validate a machine learning model to early predict the potential life-threatening conditions of SFTS. Methods The clinical presentation, demographic information, and laboratory parameters from 327 patients with SFTS at admission in three large tertiary hospitals in Jiangsu, China between 2010 to 2022 are retrieved. We establish a reservoir computing with boosted topology (RC–BT) algorithm to obtain the models’ predictions of the encephalitis and mortality of patients with SFTS. The prediction performances of encephalitis and mortality are further tested and validated. Finally, we compare our RC–BT model with the other traditional machine learning algorithms including Lightgbm, support vector machine (SVM), Xgboost, Decision Tree, and Neural Network (NN). Results For the prediction of encephalitis among patients with SFTS, nine parameters are selected with equal weight, namely calcium, cholesterol, muscle soreness, dry cough, smoking history, temperature at admission, troponin T, potassium, and thermal peak. The accuracy for the validation cohort by the RC–BT model is 0.897 [95% confidence interval (CI) 0.873–0.921]. The sensitivity and negative predictive value (NPV) of the RC–BT model are 0.855 (95% CI 0.824–0.886) and 0.904 (95% CI 0.863–0.945), respectively. Area under curve of the RC–BT model for the validation cohort is 0.899 (95% CI 0.882–0.916). For the prediction of fatality among patients with SFTS, seven parameters are selected with equal weight, namely calcium, cholesterol, history of drinking, headache, field contact, potassium, and dyspnea. The accuracy of the RC–BT model is 0.903 (95% CI 0.881–0.925). The sensitivity and NPV of the RC–BT model are 0.913 (95% CI 0.902–0.924) and 0.946 (95% CI 0.917–0.975), respectively. The area under curve is 0.917 (95% CI 0.902–0.932). Importantly, the RC–BT models outperform the other artificial intelligence-based algorithms in both prediction tasks. Conclusions Our two RC–BT models of SFTS encephalitis and fatality demonstrate high area under curves, specificity, and NPV, with nine and seven routine clinical parameters, respectively. Our models can not only greatly improve the early prognosis accuracy of SFTS, but can also be widely applied in underdeveloped areas with limited medical resources. Supplementary Information The online version contains supplementary material available at 10.1007/s40121-023-00808-y.

show abstract

Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): a retrospective deep learning study

Cited by 28 publications

References 26 publications

The Evaluation of a Clinical Decision Support Tool Using Natural Language Processing to Screen Hospitalized Adults for Unhealthy Substance Use: Protocol for a Quasi-Experimental Design

The Evaluation of a Clinical Decision Support Tool Using Natural Language Processing to Screen Hospitalized Adults for Unhealthy Substance Use: Protocol for a Quasi-Experimental Design

Towards Data-Driven Artificial Intelligence Models for Monitoring, Modelling and Predicting Illicit Substance Use

A Reservoir Computing with Boosted Topology Model to Predict Encephalitis and Mortality for Patients with Severe Fever with Thrombocytopenia Syndrome: A Retrospective Multicenter Study

Contact Info

Product

Resources

About