Extracting postmarketing adverse events from safety reports in the vaccine adverse event reporting system (VAERS) using deep learning

Abstract

Objective Automated analysis of vaccine postmarketing surveillance narrative reports is important to understand the progression of rare but severe vaccine adverse events (AEs). This study implemented and evaluated state-of-the-art deep learning algorithms for named entity recognition to extract nervous system disorder-related events from vaccine safety reports. Materials and Methods We collected Guillain-Barré syndrome (GBS) related influenza vaccine safety reports from the Vaccine Adverse Event Reporting System (VAERS) from 1990 to 2016. VAERS reports were selected and manually annotated with major entities related to nervous system disorders, including, investigation, nervous_AE, other_AE, procedure, social_circumstance, and temporal_expression. A variety of conventional machine learning and deep learning algorithms were then evaluated for the extraction of the above entities. We further pretrained domain-specific BERT (Bidirectional Encoder Representations from Transformers) using VAERS reports (VAERS BERT) and compared its performance with existing models. Results and Conclusions Ninety-one VAERS reports were annotated, resulting in 2512 entities. The corpus was made publicly available to promote community efforts on vaccine AEs identification. Deep learning-based methods (eg, bi-long short-term memory and BERT models) outperformed conventional machine learning-based methods (ie, conditional random fields with extensive features). The BioBERT large model achieved the highest exact match F-1 scores on nervous_AE, procedure, social_circumstance, and temporal_expression; while VAERS BERT large models achieved the highest exact match F-1 scores on investigation and other_AE. An ensemble of these 2 models achieved the highest exact match microaveraged F-1 score at 0.6802 and the second highest lenient match microaveraged F-1 score at 0.8078 among peer models.

Text Mining of Adverse Events in Clinical Trials: Deep Learning Approach

“…In the latter approach, we were dealing with a sequence labeling problem where the boundaries of a token sequence that referred to an adverse event needed to be determined. This is how Du et al [32] approached the extraction of adverse events from safety reports by framing it as the NER problem and fine-tuning BERT for this task. We reimplemented and cross-validated their approach on our data set to establish the baseline.…”

“…BERT is based on an encoder-decoder NN architecture, which can not only be used to generate word embeddings but can also be fine-tuned and further trained for various text mining tasks. For example, it has been used to model adverse event extraction as a named entity recognition (NER) task [11,32]. The topics of word embedding and BERT, in particular, will be revisited later in this paper in the context of motivating and describing our own approach to this problem.…”

Supervised Machine Learning-Based Decision Support for Signal Validation Classification

“…There is reinforced interest and focus in research for the use of machine learning (ML) and artificial intelligence in a growing number of pharmacovigilance processes [ 4 ], including decision support and automation in the processing and reporting of Individual Case Safety Reports (ICSRs) [ 5 – 7 ], identification of adverse events or other medical concepts from spontaneous reports or social media supported by natural language processing [ 8 – 10 ], and adverse event prediction for personalized medicine [ 11 ]. Efforts are also increasing within pharmacovigilance research to support the signal detection process using ML approaches [ 12 – 14 ].…”

Usefulness of Vaccine Adverse Event Reporting System for Machine-Learning Based Vaccine Research: A Case Study for COVID-19 Vaccines

“…VAERS datasets have been used in various studies for recommendations and proactive strategies for regulatory bodies (CDC and FDA) [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ]. To date, only limited studies have comprehensively focused on the protocols to be followed when VAERS datasets are used for statistical analyses ( Supplementary Materials—Section S1 ).…”