Pre-trained models, data augmentation, and ensemble learning for biomedical information extraction and document classification

Abstract

Large volumes of publications are being produced in biomedical sciences nowadays with ever-increasing speed. To deal with the large amount of unstructured text data, effective natural language processing (NLP) methods need to be developed for various tasks such as document classification and information extraction. BioCreative Challenge was established to evaluate the effectiveness of information extraction methods in biomedical domain and facilitate their development as a community-wide effort. In this paper, we summarize our work and what we have learned from the latest round, BioCreative Challenge VII, where we participated in all five tracks. Overall, we found three key components for achieving high performance across a variety of NLP tasks: (1) pre-trained NLP models; (2) data augmentation strategies and (3) ensemble modelling. These three strategies need to be tailored towards the specific tasks at hands to achieve high-performing baseline models, which are usually good enough for practical applications. When further combined with task-specific methods, additional improvements (usually rather small) can be achieved, which might be critical for winning competitions. Database URL: https://doi.org/10.1093/database/baac066

Artificial Intelligence for Quantitative Modeling in Drug Discovery and Development: An Innovation and Quality Consortium Perspective on Use Cases and Best Practices

“…Santos et al 87 used a KG to interpret clinical proteomics data for drug target identification and drug repurposing. Erdengasileng et al 88 proposed an approach to identify potential drug-drug interactions with high accuracy. Zhang et al 89 developed MatchMixeR, a cross-platform normalization method for gene expression data integration to identify new drug targets and potential drug combinations.…”

“…used a KG to interpret clinical proteomics data for drug target identification and drug repurposing. Erdengasileng et al 88 . proposed an approach to identify potential drug–drug interactions with high accuracy.…”

Adding Linguistic Information to Transformer Models Improves Biomedical Event Detection?

“…For this purpose, BERT, BioBERT, SciBERT, Pub-MedBERT, and BioMedRoBERTa are fine-tuned using two different classifiers, a linear layer and a Bidirectional Long Short Term Memory (Bi-LSTM) layer, to detect biomedical event triggers. These BERT variants have been chosen for comparison because they share the same BERT architecture but have previously been pretrained using different data in the biomedical and/or general domain [7]- [9]. The models are learned using seven manually annotated data sets merged together.…”