Pharmacovigilance with Transformers: A Framework to Detect Adverse Drug Reactions Using BERT Fine-Tuned with FARM

Abstract

Adverse drug reactions (ADRs) are the undesirable effects associated with the use of a drug due to some pharmacological action of the drug. During the last few years, social media has become a popular platform where people discuss their health problems and, therefore, has become a popular source to share information related to ADR in the natural language. This paper presents an end-to-end system for modelling ADR detection from the given text by fine-tuning BERT with a highly modular Framework for Adapting Representation Models (FARM). BERT overcame the predominant neural networks bringing remarkable performance gains. However, training BERT is a computationally expensive task which limits its usage for production environments and makes it difficult to determine the most important hyperparameters for the downstream task. Furthermore, developing an end-to-end ADR extraction system comprising two downstream tasks, i.e., text classification for filtering text containing ADRs and extracting ADR mentions from the classified text, is also challenging. The framework used in this work, FARM-BERT, provides support for multitask learning by combining multiple prediction heads which makes training of the end-to-end systems easier and computationally faster. In the proposed model, one prediction head is used for text classification and the other is used for ADR sequence labeling. Experiments are performed on Twitter, PubMed, TwiMed-Twitter, and TwiMed-PubMed datasets. The proposed model is compared with the baseline models and state-of-the-art techniques, and it is shown that it yields better results for the given task with the F -scores of 89.6%, 97.6%, 84.9%, and 95.9% on Twitter, PubMed, TwiMed-Twitter, and TwiMed-PubMed datasets, respectively. Moreover, training time and testing time of the proposed model are compared with BERT’s, and it is shown that the proposed model is computationally faster than BERT.

Users’ Reactions to Announced Vaccines Against COVID-19 Before Marketing in France: Analysis of Twitter Posts

“…Tweets are more difficult to classify due to their characteristics: They are limited to 280 characters but can be much shorter; they are not necessarily written with correct syntax; and they can contain abbreviations, misspellings, and slang words [ 44 ]. We also observed that the results of predictions are generally significantly better for correctly written texts such as PubMed abstracts, than with tweets, for example [ 45 ].…”

“…This confirms that data science is an empirical discipline in which only experiments make it possible to determine the most appropriate approach in a defined context and that there is no method that makes it possible to confidently improve the results. In some cases, multitask learning made it possible to improve the results [ 45 , 62 , 63 ]. However, previous studies, such as that by Rodriguez et al [ 64 ], observed that multitask learning did not improve the results in general.…”

Large Language Models to process, analyze, and synthesize biomedical texts – a scoping review

“…[30] used contextual information in extracting long distance adverse drug events from clinical notes with BERT models. There were also overlaps between domains, such as work that introduced a system for detecting adverse drug reactions from text on social media by fine-tuning BERT [31].…”

Large Language Models for Detecting Body Weight Changes as Side Effects of Antidepressants in User-Generated Online Content

“…Weight loss seems more difficult to classify and can be confused with weight gain - perhaps because these two classes share very similar syntactic elements. If an expert review of identified signals will likely remain essential for assessing a drug’s causality in the occurrence of adverse effects, it will be interesting to investigate how these models can aid in contextualizing the findings to support causality determination (47). Because online comments are often written with little contextual information, it is difficult to establish a causal relationship between drug use and the occurrence of an event.…”