Multi-label classification of symptom terms from free-text bilingual adverse drug reaction reports using natural language processing

Abstract

Allergic reactions to medication range from mild to severe or even life-threatening. Proper documentation of patient allergy information is critical for safe prescription, avoiding drug interactions, and reducing healthcare costs. Allergy information is regularly obtained during the medical interview, but is often poorly documented in electronic health records (EHRs). While many EHRs allow for structured adverse drug reaction (ADR) reporting, a free-text entry is still common. The resulting information is neither interoperable nor easily reusable for other applications, such as clinical decision support systems and prescription alerts. Current approaches require pharmacists to review and code ADRs documented by healthcare professionals. Recently, the effectiveness of machine algorithms in natural language processing (NLP) has been widely demonstrated. Our study aims to develop and evaluate different NLP algorithms that can encode unstructured ADRs stored in EHRs into institutional symptom terms. Our dataset consists of 79,712 pharmacist-reviewed drug allergy records. We evaluated three NLP techniques: Naive Bayes—Support Vector Machine (NB-SVM), Universal Language Model Fine-tuning (ULMFiT), and Bidirectional Encoder Representations from Transformers (BERT). We tested different general-domain pre-trained BERT models, including mBERT, XLM-RoBERTa, and WanchanBERTa, as well as our domain-specific AllergyRoBERTa, which was pre-trained from scratch on our corpus. Overall, BERT models had the highest performance. NB-SVM outperformed ULMFiT and BERT for several symptom terms that are not frequently coded. The ensemble model achieved an exact match ratio of 95.33%, a F1 score of 98.88%, and a mean average precision of 97.07% for the 36 most frequently coded symptom terms. The model was then further developed into a symptom term suggestion system and achieved a Krippendorff’s alpha agreement coefficient of 0.7081 in prospective testing with pharmacists. Some degree of automation could both accelerate the availability of allergy information and reduce the efforts for human coding.

The Indonesian Young-Adult Attachment (IYAA): An audio-video dataset for behavioral young-adult attachment assessment

“…It is a measurement that compares the observed disagreement with the expected disagreement. We choose this approach since it is reliable for handling multi-label data, e.g., [ 19 , 20 ]. We obtained Krippendorff's alpha agreement of 0.9099, 0.6489, and 0.8115 on exposure-only, response-only, and exposure-response, respectively.…”

Optimizing classification of diseases through language model analysis of symptoms

“…Chaichulee et al 22 evaluated three NLP techniques—Naive Bayes-Support Vector Machine (NB-SVM), Universal Language Model Fine-tuning (ULMFiT), and various pre-trained BERT models including mBERT, XLM-RoBERTa, WanchanBERTa, and a domain-specific AllergyRoBERTa model trained on a dataset of 79,712 drug allergy records reviewed by pharmacists—to identify symptom terms from clinical notes, finding that while the BERT models generally demonstrated the highest performance, the NB-SVM model outperformed ULMFiT and BERT for less frequently coded symptoms. An ensemble model combining the different algorithms achieved strong results with 95.33% exact match ratio, 98.88% F1 score, and 97.07% mean average precision for the 36 most frequent symptoms, and this developed model was further enhanced into a symptom term suggestion system that tested well in prospective pharmacist trials with a 0.7081 Krippendorff's alpha agreement coefficient, indicating reasonably high agreement between the model's suggestions and pharmacist assessments.…”

Artificial Intelligence: Exploring the Future of Innovation in Allergy Immunology

“…Having a penicillin allergy label is associated with higher healthcare resource utilization, emphasizing the need for accurate diagnosis and labeling [ 59 ]. ML has been used to risk stratify, and potentially de-label penicillin allergy with medical record analysis playing a key role in addressing drug allergy at a population level [ 60 , 61 , 62 •, 63 , 64 ].…”

“…Inaccuracies in documented drug allergies in the EHR are common and can be classified using AI. In one example, NLP algorithms were found to be more accurate in identifying and classifying drug reactions than ICD diagnostic codes review or equally as accurate to manual classification by pharmacists [ 62 •, 63 ]. Even patients who have demonstrated tolerance to a prior listed allergy with a drug challenge continue to be labeled as allergic in the EHR.…”