Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes

Boag, William; Kovaleva, Olga; McCoy, Thomas H.; Rumshisky, Anna; Szolovits, Peter; Perlis, Roy H.

doi:10.1038/s41398-020-01104-w

Cited by 16 publications

(9 citation statements)

References 13 publications

(15 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, using Natural Language Processing methods for feature construction for predictive task using the ClinicalTrials.gov data is an untapped area. We expect the combination of free-text based features and structured data could improve the predictive performance for enrollment rate, as demonstrated in prior studies in other data domain [ 37 – 39 ].…”

Section: Discussionmentioning

confidence: 59%

Prediction of clinical trial enrollment rates

Bieganek

Aliferis

2022

PLoS ONE

View full text Add to dashboard Cite

Clinical trials represent a critical milestone of translational and clinical sciences. However, poor recruitment to clinical trials has been a long standing problem affecting institutions all over the world. One way to reduce the cost incurred by insufficient enrollment is to minimize initiating trials that are most likely to fall short of their enrollment goal. Hence, the ability to predict which proposed trials will meet enrollment goals prior to the start of the trial is highly beneficial. In the current study, we leveraged a data set extracted from ClinicalTrials.gov that consists of 46,724 U.S. based clinical trials from 1990 to 2020. We constructed 4,636 candidate predictors based on data collected by ClinicalTrials.gov and external sources for enrollment rate prediction using various state-of-the-art machine learning methods. Taking advantage of a nested time series cross-validation design, our models resulted in good predictive performance that is generalizable to future data and stable over time. Moreover, information content analysis revealed the study design related features to be the most informative feature type regarding enrollment. Compared to the performance of models built with all features, the performance of models built with study design related features is only marginally worse (AUC = 0.78 ± 0.03 vs. AUC = 0.76 ± 0.02). The results presented can form the basis for data-driven decision support systems to assess whether proposed clinical trials would likely meet their enrollment goal.

show abstract

Section: Discussionmentioning

confidence: 59%

Prediction of clinical trial enrollment rates

Bieganek

Aliferis

2022

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…Furthermore, the LDA method is relatively straightforward to understand since it reflects semantic characteristics compared to other machine learning techniques using black-box algorithms and insufficient transparency ( 37 ). Thanks to these advantages, prediction models using the LDA method have been used in several studies ( 38 , 39 ).…”

Section: Discussionmentioning

confidence: 99%

Psychosis Relapse Prediction Leveraging Electronic Health Records Data and Natural Language Processing Enrichment Methods

et al. 2022

View full text Add to dashboard Cite

BackgroundIdentifying patients at a high risk of psychosis relapse is crucial for early interventions. A relevant psychiatric clinical context is often recorded in clinical notes; however, the utilization of unstructured data remains limited. This study aimed to develop psychosis-relapse prediction models using various types of clinical notes and structured data.MethodsClinical data were extracted from the electronic health records of the Ajou University Medical Center in South Korea. The study population included patients with psychotic disorders, and outcome was psychosis relapse within 1 year. Using only structured data, we developed an initial prediction model, then three natural language processing (NLP)-enriched models using three types of clinical notes (psychological tests, admission notes, and initial nursing assessment) and one complete model. Latent Dirichlet Allocation was used to cluster the clinical context into similar topics. All models applied the least absolute shrinkage and selection operator logistic regression algorithm. We also performed an external validation using another hospital database.ResultsA total of 330 patients were included, and 62 (18.8%) experienced psychosis relapse. Six predictors were used in the initial model and 10 additional topics from Latent Dirichlet Allocation processing were added in the enriched models. The model derived from all notes showed the highest value of the area under the receiver operating characteristic (AUROC = 0.946) in the internal validation, followed by models based on the psychological test notes, admission notes, initial nursing assessments, and structured data only (0.902, 0.855, 0.798, and 0.784, respectively). The external validation was performed using only the initial nursing assessment note, and the AUROC was 0.616.ConclusionsWe developed prediction models for psychosis relapse using the NLP-enrichment method. Models using clinical notes were more effective than models using only structured data, suggesting the importance of unstructured data in psychosis prediction.

show abstract

“…LDA topic modeling has been used for a variety of NLP tasks [63,64] (although it can also be used on other high-dimension data) such as text classification and filtering [65]. LDA topic modeling has been applied to the unstructured notes of EHRs to describe clinical groups [104][105][106][107][108] and predicting outcomes [109][110][111][112][113][114][115][116]. We were unable to find published instances of LDA topic-modeling applications for AE detection.…”

Section: Comparison Of the Shakespeare Methods To Other Applications Of Lda Topic Modelingmentioning

confidence: 99%

Finding Potential Adverse Events in the Unstructured Text of Electronic Health Care Records: Development of the Shakespeare Method

Bright¹,

Rankin²,

Dowdy³

et al. 2021

JMIRx Med

View full text Add to dashboard Cite

Background Big data tools provide opportunities to monitor adverse events (patient harm associated with medical care) (AEs) in the unstructured text of electronic health care records (EHRs). Writers may explicitly state an apparent association between treatment and adverse outcome (“attributed”) or state the simple treatment and outcome without an association (“unattributed”). Many methods for finding AEs in text rely on predefining possible AEs before searching for prespecified words and phrases or manual labeling (standardization) by investigators. We developed a method to identify possible AEs, even if unknown or unattributed, without any prespecifications or standardization of notes. Our method was inspired by word-frequency analysis methods used to uncover the true authorship of disputed works credited to William Shakespeare. We chose two use cases, “transfusion” and “time-based.” Transfusion was chosen because new transfusion AE types were becoming recognized during the study data period; therefore, we anticipated an opportunity to find unattributed potential AEs (PAEs) in the notes. With the time-based case, we wanted to simulate near real-time surveillance. We chose time periods in the hope of detecting PAEs due to contaminated heparin from mid-2007 to mid-2008 that were announced in early 2008. We hypothesized that the prevalence of contaminated heparin may have been widespread enough to manifest in EHRs through symptoms related to heparin AEs, independent of clinicians’ documentation of attributed AEs. Objective We aimed to develop a new method to identify attributed and unattributed PAEs using the unstructured text of EHRs. Methods We used EHRs for adult critical care admissions at a major teaching hospital (2001-2012). For each case, we formed a group of interest and a comparison group. We concatenated the text notes for each admission into one document sorted by date, and deleted replicate sentences and lists. We identified statistically significant words in the group of interest versus the comparison group. Documents in the group of interest were filtered to those words, followed by topic modeling on the filtered documents to produce topics. For each topic, the three documents with the maximum topic scores were manually reviewed to identify PAEs. Results Topics centered around medical conditions that were unique to or more common in the group of interest, including PAEs. In each use case, most PAEs were unattributed in the notes. Among the transfusion PAEs was unattributed evidence of transfusion-associated cardiac overload and transfusion-related acute lung injury. Some of the PAEs from mid-2007 to mid-2008 were increased unattributed events consistent with AEs related to heparin contamination. Conclusions The Shakespeare method could be a useful supplement to AE reporting and surveillance of structured EHR data. Future improvements should include automation of the manual review process.

show abstract

Hard for humans, hard for machines: predicting readmission after psychiatric hospitalization using narrative notes

Cited by 16 publications

References 13 publications

Prediction of clinical trial enrollment rates

Prediction of clinical trial enrollment rates

Psychosis Relapse Prediction Leveraging Electronic Health Records Data and Natural Language Processing Enrichment Methods

Finding Potential Adverse Events in the Unstructured Text of Electronic Health Care Records: Development of the Shakespeare Method

Contact Info

Product

Resources

About