Chemical–protein relation extraction with ensembles of carefully tuned pretrained language models

Abstract

The identification of chemical–protein interactions described in the literature is an important task with applications in drug design, precision medicine and biotechnology. Manual extraction of such relationships from the biomedical literature is costly and often prohibitively time-consuming. The BioCreative VII DrugProt shared task provides a benchmark for methods for the automated extraction of chemical–protein relations from scientific text. Here we describe our contribution to the shared task and report on the achieved results. We define the task as a relation classification problem, which we approach with pretrained transformer language models. Upon this basic architecture, we experiment with utilizing textual and embedded side information from knowledge bases as well as additional training data to improve extraction performance. We perform a comprehensive evaluation of the proposed model and the individual extensions including an extensive hyperparameter search leading to 2647 different runs. We find that ensembling and choosing the right pretrained language model are crucial for optimal performance, whereas adding additional data and embedded side information did not improve results. Our best model is based on an ensemble of 10 pretrained transformers and additional textual descriptions of chemicals taken from the Comparative Toxicogenomics Database. The model reaches an F1 score of 79.73% on the hidden DrugProt test set and achieves the first rank out of 107 submitted runs in the official evaluation. Database URL: https://github.com/leonweber/drugprot

Integrating heterogeneous knowledge graphs into drug–drug interaction extraction from the literature

“…If we employ an ensemble method for the DrugProt task, e.g. Weber et al (2021) and Luo et al (2021) , we can expect better performance; however, it is beyond the scope of this article.…”

Mining drug-target interactions from biomedical literature using chemical and gene descriptions-based ensemble transformer model

“…We approached the drug-target relation classification task as a multi-label problem similar to [5]. However, we employed various setup configurations, including the use of Focal Loss instead of binary cross-entropy loss.…”

“…Figure 1 We conducted experiments using various model configurations, and subsequently linked chemical and gene to their ontologies. To ensure consistency, we adopted the same unique gene identifiers as in [5] for both the training and development datasets. These identifiers were normalized using a BioSync model [25] for chemicals, leveraging the comprehensive BioCreative V CDR (BC5CDR) dataset [26], and for proteins, utilizing the BioCreative II Gene Normalization (BC2GN) dataset [27].…”

“…For the protein annotation, we obtained the descriptions from the UniProt database. In our study, we utilized the same chemical descriptions as in [5], which were derived from the Comparative Toxicogenomics Database (CTD). These descriptions were obtained from the Definition field of the CTD's chemical vocabulary, specifically taking the first sentence of the definition.…”

“…Most of the participants in the DrugProt challenge applied the most recent pre-trained models like BERT to train on the domain of drug-target interactions. For example, in [5], authors leveraged ensembles of pre-trained language models such as RoBERTa-large [6], using chemical descriptions to achieve an F1 score of 79.73 as the best model in the DrugProt challenge. More recently, authors in [7] employed a sequence labelling framework, achieving a state-of-the-art F1-score of 80.0 on the test set.…”