BertSRC: transformer-based semantic relation classification

Abstract

The relationship between biomedical entities is complex, and many of them have not yet been identified. For many biomedical research areas including drug discovery, it is of paramount importance to identify the relationships that have already been established through a comprehensive literature survey. However, manually searching through literature is difficult as the amount of biomedical publications continues to increase. Therefore, the relation classification task, which automatically mines meaningful relations from the literature, is spotlighted in the field of biomedical text mining. By applying relation classification techniques to the accumulated biomedical literature, existing semantic relations between biomedical entities that can help to infer previously unknown relationships are efficiently grasped. To develop semantic relation classification models, which is a type of supervised machine learning, it is essential to construct a training dataset that is manually annotated by biomedical experts with semantic relations among biomedical entities. Any advanced model must be trained on a dataset with reliable quality and meaningful scale to be deployed in the real world and can assist biologists in their research. In addition, as the number of such public datasets increases, the performance of machine learning algorithms can be accurately revealed and compared by using those datasets as a benchmark for model development and improvement. In this paper, we aim to build such a dataset. Along with that, to validate the usability of the dataset as training data for relation classification models and to improve the performance of the relation extraction task, we built a relation classification model based on Bidirectional Encoder Representations from Transformers (BERT) trained on our dataset, applying our newly proposed fine-tuning methodology. In experiments comparing performance among several models based on different deep learning algorithms, our model with the proposed fine-tuning methodology showed the best performance. The experimental results show that the constructed training dataset is an important information resource for the development and evaluation of semantic relation extraction models. Furthermore, relation extraction performance can be improved by integrating our proposed fine-tuning methodology. Therefore, this can lead to the promotion of future text mining research in the biomedical field.

FA-RCNet: A Fused Feature Attention Network for Relationship Classification

“…BERT-GMAN [11] proposed a relationship extraction model based on BERTgated multi-window attention network, which achieved good results in the Semeval-2010 Task 8 dataset. BertSRC [42] proposed to extract the entity relationship in the medical field based on BERT and achieved good results in the data set of the proprietary field. AugFake-BERT [43] proposes a BERT-based data augmentation model.…”

Unsupervised literature mining approaches for extracting relationships pertaining to habitats and reproductive conditions of plant species

“…These DNN models learn higher-order, abstract feature representations from sentences. With the emergence of DNNs, models that employ neural architectures such as convolutional neural networks (CNNs; Liu et al, 2013;dos Santos et al, 2015), recurrent neural networks (RNN; Zhang and Wang, 2015;Vu et al, 2016), graph convolutional networks (GCN; Zhu et al, 2019), attention-based neural networks (Wang et al, 2016;Xiao and Liu, 2016), and transformer-based language models (Vaswani et al, 2017;Lee et al, 2022) have been utilized for RE tasks. Like traditional ML-based models, DNN-based models learn features from data.…”

A hybrid algorithm for clinical decision support in precision medicine based on machine learning

“…Two types of BERT models, BERT BASE and BERT LARGE , are available [38]. Some articles covering various related modifications of BERT can be found in [39][40][41][42].…”