Identifying Symptom Information in Clinical Notes Using Natural Language Processing

Abstract

Background: Symptoms are a core concept of nursing interest. Large-scale secondary data reuse of notes in electronic health records (EHRs) has the potential to increase the quantity and quality of symptom research. However, the symptom language used in clinical notes is complex. A need exists for methods designed specifically to identify and study symptom information from EHR notes.Objectives: We aim to describe a method that combines standardized vocabularies, clinical expertise, and natural language processing to generate comprehensive symptom vocabularies and identify symptom information in EHR notes. We piloted this method with five diverse symptom concepts: constipation, depressed mood, disturbed sleep, fatigue, and palpitations.Methods: First, we obtained synonym lists for each pilot symptom concept from the Unified Medical Language System. Then, we used two large bodies of text (clinical notes from Columbia University Irving Medical Center and PubMed abstracts containing Medical Subject Headings or key words related to the pilot symptoms) to further expand our initial vocabulary of synonyms for each pilot symptom concept. We used NimbleMiner, an open-source natural language processing tool, to accomplish these tasks and evaluated NimbleMiner symptom identification performance by comparison to a manually annotated set of nurse-and physician-authored common EHR note types.Results: Compared to the baseline Unified Medical Language System synonym lists, we identified up to 11 times more additional synonym words or expressions, including abbreviations, misspellings, and unique multiword combinations, for each symptom concept. Natural language processing system symptom identification performance was excellent.Discussion: Using our comprehensive symptom vocabularies and NimbleMiner to label symptoms in clinical notes produced excellent performance metrics. The ability to extract symptom information from EHR notes in an accurate and scalable manner has the potential to greatly facilitate symptom science research.

Characterizing shared and distinct symptom clusters in common chronic conditions through natural language processing of nursing notes

“…Our team previously generated and validated comprehensive symptom vocabularies using a method that combined standardized vocabularies from the Unified Medical Language System, clinical expertise of nurse clinician scientists, and NLP (Koleck et al, 2020). Based on this earlier work, we used NimbleMiner, an open-source NLP RStudio application (https://github.com/mtopaz/NimbleMiner), to identify the presence of symptoms in nursing notes at the note level.…”

“…Natural language processing (NLP), which is "any computer-based algorithm that handles, augments, and transforms natural language so that it can be represented for computation" (Yim et al, 2016), can be used to study symptom information from text-based nursing notes (Koleck et al, 2019). Our team recently developed a method that combines standardized vocabularies, clinical expertise, and NLP to generate comprehensive symptom vocabularies and identify and extract symptom information in EHR notes in an accurate and scalable manner (Koleck et al, 2020).…”

Using Symptoms and Healthcare Encounters to Capture a Rare Disease: A Study of Clinical Notes of the Alpha‐Gal Meat Allergy

“…Given this understanding of AGS, this study employed a clinical textual analysis for understanding the symptoms and environmental factors of AGS. Electronic clinical notes have been used to help understand medical symptoms (Koleck et al, 2020), and to help with the diagnosis of rare disease with some assistance of computer algorithms (Colbaugh et al, 2018; Dong et al, 2021; Sheikhalishahi et al, 2019). However, researching clinical notes of AGS can present a unique challenge because: first, manually going through clinical notes is possible and necessary due to the limited amount of clinical notes available; in addition, medical research is also at an early stage of gaining a full picture of the diagnosis of this disease.…”

Identifying patients presenting in pain to the adult emergency department: A binary classification task and description of prevalence

“…To be generalisable to the large population that presents to EDs worldwide, a retrospective methodology that uses widely collected patient information is required. Previous authors have identified that AI approaches, especially Natural Language Processing (NLP) are useful for the classification of symptoms in electronic health records (32, 33). Early pilot studies of the application of this technology in identifying pain within triage nursing assessments have proved feasible and successful (34, 35).…”