Integration and Validation of a Natural Language Processing Machine Learning Suicide Risk Prediction Model Based on Open-Ended Interview Language in the Emergency Department

Abstract

BackgroundEmergency departments (ED) are an important intercept point for identifying suicide risk and connecting patients to care, however, more innovative, person-centered screening tools are needed. Natural language processing (NLP) -based machine learning (ML) techniques have shown promise to assess suicide risk, although whether NLP models perform well in differing geographic regions, at different time periods, or after large-scale events such as the COVID-19 pandemic is unknown.ObjectiveTo evaluate the performance of an NLP/ML suicide risk prediction model on newly collected language from the Southeastern United States using models previously tested on language collected in the Midwestern US.Method37 Suicidal and 33 non-suicidal patients from two EDs were interviewed to test a previously developed suicide risk prediction NLP/ML model. Model performance was evaluated with the area under the receiver operating characteristic curve (AUC) and Brier scores.ResultsNLP/ML models performed with an AUC of 0.81 (95% CI: 0.71–0.91) and Brier score of 0.23.ConclusionThe language-based suicide risk model performed with good discrimination when identifying the language of suicidal patients from a different part of the US and at a later time period than when the model was originally developed and trained.

Evaluation of a Model to Target High-risk Psychiatric Inpatients for an Intensive Postdischarge Suicide Prevention Intervention

“…Our finding that NLP and LexisNexis SDOH variables improve model prediction accuracy is broadly consistent with prior studies of NLP and SDOH variables predicting suicide, although to our knowledge, no prior studies investigated these associations among postdischarge psychiatric inpatients. Four results about these predictors are especially noteworthy in terms of potential clinical implications.…”

“…7 Based on the strength of that model, an experimental trial was funded for suicide prevention with intensive postdischarge case management of highrisk patients. 8 However, it was decided that further study was needed before implementing the trial to determine if model accuracy could be improved by adding 2 types of predictors found to be important in other suicide prediction models: (1) predictors extracted from clinical notes using natural language processing (NLP) methods 9 and (2) indicators of patientlevel social determinants of health (SDOH) extracted from public records. 10 Results of this investigation are presented here along with an analysis of the net benefit (NB) of the expanded model across a range of plausible decision thresholds.…”

User Feedback on the Use of a Natural Language Processing Application to Screen for Suicide Risk in the Emergency Department

“…Recent work has demonstrated the successful integration of a suicide risk screening interview to collect language data for NLP analysis from patients in two emergency departments (ED) of a large healthcare system. 18 Results from this study suggested that ML/NLP models performed well in identifying patients that came to the ED for suicide risk in an area of the country where speech dialects vary from language samples used in the original development of the technology. 16 However, little is known about the clinician's perspective of how a qualitative brief interview suicide risk screening tool to collect language data for NLP integrates into an ED workflow.…”

Application of Natural Language Processing (NLP) in Detecting and Preventing Suicide Ideation: A Systematic Review

“…Estimating the likelihood of multiple suicide attempts is largely left to clinical judgement in the Emergency Department, where suicidal patients often appear (J.P. et al, 2008). Early recognition of selfharm presentations to emergency departments (ED) may result in more prompt suicide ideation care.In a research conducted by(Cohen et al, 2022) it was investigated whether the interview approach to obtain linguistic data for the NLP/ML model might be implemented in two emergency departments in the South-eastern United States. In their research, interviews were conducted with 37 suicidal and 33 non-suicidal patients from two emergency departments to evaluate the NLP/ML suicide risk prediction model.…”

“…Area under the receiver operating characteristic curve (AUC) and Brier scores were used to assess the model's performance.The research demonstrates that it is viable to integrate technology and methods to gather linguistic data for a suicide risk prediction model into the emergency department workflow. In addition, a fast interview with patients may be used efficiently in the emergency department, and NLP/ML models can reliably predict the patient's suicide risk based on their comments.Similar to(Cohen et al, 2022),(Pestian et al, 2016) created a prospective clinical trial to examine the claim that machine learning techniques may distinguish between suicidal and non-suicidal people by listening to their conversations. NLP and semi supervised machine learning techniques were used to record and evaluate the discussions of 30 suicidal teenagers and 30 matched controls using questionnaires and interviews as the data collection tools.…”