Generating contextual embeddings for emergency department chief complaints

Abstract

Objective We learn contextual embeddings for emergency department (ED) chief complaints using Bidirectional Encoder Representations from Transformers (BERT), a state-of-the-art language model, to derive a compact and computationally useful representation for free-text chief complaints. Materials and methods Retrospective data on 2.1 million adult and pediatric ED visits was obtained from a large healthcare system covering the period of March 2013 to July 2019. A total of 355 497 (16.4%) visits from 65 737 (8.9%) patients were removed for absence of either a structured or unstructured chief complaint. To ensure adequate training set size, chief complaint labels that comprised less than 0.01%, or 1 in 10 000, of all visits were excluded. The cutoff threshold was incremented on a log scale to create seven datasets of decreasing sparsity. The classification task was to predict the provider-assigned label from the free-text chief complaint using BERT, with Long Short-Term Memory (LSTM) and Embeddings from Language Models (ELMo) as baselines. Performance was measured as the Top-k accuracy from k = 1:5 on a hold-out test set comprising 5% of the samples. The embedding for each free-text chief complaint was extracted as the final 768-dimensional layer of the BERT model and visualized using t-distributed stochastic neighbor embedding (t-SNE). Results The models achieved increasing performance with datasets of decreasing sparsity, with BERT outperforming both LSTM and ELMo. The BERT model yielded Top-1 accuracies of 0.65 and 0.69, Top-3 accuracies of 0.87 and 0.90, and Top-5 accuracies of 0.92 and 0.94 on datasets comprised of 434 and 188 labels, respectively. Visualization using t-SNE mapped the learned embeddings in a clinically meaningful way, with related concepts embedded close to each other and broader types of chief complaints clustered together. Discussion Despite the inherent noise in the chief complaint label space, the model was able to learn a rich representation of chief complaints and generate reasonable predictions of their labels. The learned embeddings accurately predict provider-assigned chief complaint labels and map semantically similar chief complaints to nearby points in vector space. Conclusion Such a model may be used to automatically map free-text chief complaints to structured fields and to assist the development of a standardized, data-driven ontology of chief complaints for healthcare institutions.

Sampling and ranking spatial transcriptomics data embeddings to identify tissue architecture

“…To set a comprehensive comparison metric, we used the quality of the top 5 embeddings rather than the top 1 in the ranking to assess an evaluation method. The top 5 evaluation is also applied to other research problems (Xiong et al, 2017;Chang et al, 2020). The top 5 in the ranking represents the first five high-quality embeddings among all input embeddings identified by our evaluation method.…”

Applications of Natural Language Processing at Emergency Department Triage: A Systematic Review

“…There were 18 retrospective studies. 17,18,30,[34][35][36][37][38][39][40][41][42][43][44][45][46][47][48] One study reported their ML model was developed using retrospective data then validated using prospective data. 49 All used observational cohort designs.…”

“…36,37 Prediction of provider-assigned chief complaint NLP models and multimodal models incorporating NLP were able to accurately map freetext history of presenting complaint to structured chief complaints. 42,49…”

Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review

“…Although the TF-IDF model outperformed the BERT-based models here, the authors suspect that this result is in part due to the short length of the median text entry; the TF-IDF model robustness remains questionable. Chang et al [28] have also worked on applying a pre-trained BERT model to chief complaint extraction. They derived contextual embeddings to predict providerassigned labels, focusing more specifically on emergency department chief complaints.…”