Strategies to Address the Lack of Labeled Data for Supervised Machine Learning Training With Electronic Health Records: Case Study for the Extraction of Symptoms From Clinical Notes

Abstract

Background Automated extraction of symptoms from clinical notes is a challenging task owing to the multidimensional nature of symptom description. The availability of labeled training data is extremely limited owing to the nature of the data containing protected health information. Natural language processing and machine learning to process clinical text for such a task have great potential. However, supervised machine learning requires a great amount of labeled data to train a model, which is at the origin of the main bottleneck in model development. Objective The aim of this study is to address the lack of labeled data by proposing 2 alternatives to manual labeling for the generation of training labels for supervised machine learning with English clinical text. We aim to demonstrate that using lower-quality labels for training leads to good classification results. Methods We addressed the lack of labels with 2 strategies. The first approach took advantage of the structured part of electronic health records and used diagnosis codes (International Classification of Disease–10th revision) to derive training labels. The second approach used weak supervision and data programming principles to derive training labels. We propose to apply the developed framework to the extraction of symptom information from outpatient visit progress notes of patients with cardiovascular diseases. Results We used >500,000 notes for training our classification model with International Classification of Disease–10th revision codes as labels and >800,000 notes for training using labels derived from weak supervision. We show that the dependence between prevalence and recall becomes flat provided a sufficiently large training set is used (>500,000 documents). We further demonstrate that using weak labels for training rather than the electronic health record codes derived from the patient encounter leads to an overall improved recall score (10% improvement, on average). Finally, the external validation of our models shows excellent predictive performance and transferability, with an overall increase of 20% in the recall score. Conclusions This work demonstrates the power of using a weak labeling pipeline to annotate and extract symptom mentions in clinical text, with the prospects to facilitate symptom information integration for a downstream clinical task such as clinical decision support.

Identifying Risk Factors Associated With Lower Back Pain in Electronic Medical Record Free Text: Deep Learning Approach Using Clinical Note Annotations

“…Prior studies have explored methods for addressing the challenge of obtaining sufficient data for training [ 16 ]. To acquire clinical notes for labeling that are more likely to exhibit a minority risk factor, we used unsupervised semantic textual similarity (STS).…”

Harmonizing the Scale: An End-to-End Self-Supervised Framework for Cross-Modal Search and Retrieval in Histopathology Archives

“…As was highlighted in the introduction, the scarcity of adequately labelled data for model training is recognized as a prevailing challenge in the medical field 37 . This challenge is notably apparent in the visual modality, even though numerous dependable medical language models recently have been introduced 38,39 .…”

Weakly supervised spatial relation extraction from radiology reports

“…Fries et al 9 utilized data programming with BioBERT to classify medical entities and demonstrated comparable results to fully supervised models on multiple benchmark datasets. Very recently, Humbert-Droz et al 27 developed a data programming-based weak supervision pipeline using Snorkel to generate weak labels for identifying the presence or absence of symptoms. Moreover, in the biomedical domain, multiple studies have used the Snorkel framework for extracting chemical reaction relationships from biomedical abstracts, 28 biomedical relation extraction, 29 and filtering biomedical research articles as relevant or nonrelevant for drug repurposing in cancer.…”