Relation classification via BERT with piecewise convolution and focal loss

Duan, Xi; Zhang, Ru; Sun, Youqiang; Guan, Lei; Lin, Bingjie

doi:10.1371/journal.pone.0257092

Cited by 11 publications

(9 citation statements)

References 27 publications

(31 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The current leading NLP models such as BERT [20], GPT [21], and T5 [22] announced later are all based on this transformer block. In particular, BERT is commonly used in biomedical text mining research because it is built on multiple transformers encoder blocks, which has the advantage of compressing the sentence and mining semantic information from it [8,[23][24][25].…”

Section: Deep Learning-based Semantic Relation Classification Modelmentioning

confidence: 99%

BertSRC: transformer-based semantic relation classification

Lee

Son

Song

2022

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

The relationship between biomedical entities is complex, and many of them have not yet been identified. For many biomedical research areas including drug discovery, it is of paramount importance to identify the relationships that have already been established through a comprehensive literature survey. However, manually searching through literature is difficult as the amount of biomedical publications continues to increase. Therefore, the relation classification task, which automatically mines meaningful relations from the literature, is spotlighted in the field of biomedical text mining. By applying relation classification techniques to the accumulated biomedical literature, existing semantic relations between biomedical entities that can help to infer previously unknown relationships are efficiently grasped. To develop semantic relation classification models, which is a type of supervised machine learning, it is essential to construct a training dataset that is manually annotated by biomedical experts with semantic relations among biomedical entities. Any advanced model must be trained on a dataset with reliable quality and meaningful scale to be deployed in the real world and can assist biologists in their research. In addition, as the number of such public datasets increases, the performance of machine learning algorithms can be accurately revealed and compared by using those datasets as a benchmark for model development and improvement. In this paper, we aim to build such a dataset. Along with that, to validate the usability of the dataset as training data for relation classification models and to improve the performance of the relation extraction task, we built a relation classification model based on Bidirectional Encoder Representations from Transformers (BERT) trained on our dataset, applying our newly proposed fine-tuning methodology. In experiments comparing performance among several models based on different deep learning algorithms, our model with the proposed fine-tuning methodology showed the best performance. The experimental results show that the constructed training dataset is an important information resource for the development and evaluation of semantic relation extraction models. Furthermore, relation extraction performance can be improved by integrating our proposed fine-tuning methodology. Therefore, this can lead to the promotion of future text mining research in the biomedical field.

show abstract

Section: Deep Learning-based Semantic Relation Classification Modelmentioning

confidence: 99%

BertSRC: transformer-based semantic relation classification

Lee

Son

Song

2022

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

show abstract

“…They also leveraged the entity information in their proposed model to improve performance. In a recent study, Liu et al further extend this architecture in [29]. To capture the latent information around the target entities, the authors utilize piecewise convolution [30].…”

Section: Related Workmentioning

confidence: 99%

“…We also use another BERT-based architecture proposed in [29] with minor changes. This architecture is an extension of the previous BERT-based architecture proposed by [28].…”

Section: ) Multilingual-r-bert + Pcnnmentioning

confidence: 99%

See 1 more Smart Citation

Event-Argument Linking in Disaster Domain

et al. 2022

View full text Add to dashboard Cite

Linking event triggers with their respective arguments is an essential component for building an event extraction system. It is challenging to link event triggers with the corresponding arguments triggers when the sentence contains multiple events and arguments triggers. The task becomes even more challenging in a low-resource setup due to the unavailability of natural language processing resource and tools. In this paper, we study the event-argument linking task based on disaster event ontology in a low resource setup. We use BERT and non-BERT-based deep learning models in both monolingual and cross-lingual eventargument linking task. We also perform an ablation study of various features like position embeddings (PE), position indicator (PI), and segment ID (SI) to understand their contribution to performance improvement in non-BERT-based models. Using three different languages viz. Hindi, Bengali, and Marathi, we compare the results with multilingual BERT-based deep neural models in both monolingual and cross-lingual scenarios. We observe that the multilingual BERT-based model outperforms the best performing non-BERT-based model in cross-lingual settings. But in monolingual settings, the performance is similar in Hindi and Bengali datasets and slightly better in Marathi dataset. We choose the disaster domain due to its social implications.Our current experiments can be helpful in mining important information related to disaster events from news articles and building event knowledge graphs in low-resource languages.

show abstract

“…To avoid potential overfitting, we used an early stop when the learning rate drops below 10 -6 or 1000 epochs were exceeded. Focal loss function was applied (33,34). Note that the deep learning model used only the image information where clinical features were not included.…”

Section: Deep Learning Model Constructionmentioning

confidence: 99%

3D deep learning versus the current methods for predicting tumor invasiveness of lung adenocarcinoma based on high-resolution computed tomography images

Wang²,

Xu³

et al. 2022

Front. Oncol.

View full text Add to dashboard Cite

BackgroundDifferent pathological subtypes of lung adenocarcinoma lead to different treatment decisions and prognoses, and it is clinically important to distinguish invasive lung adenocarcinoma from preinvasive adenocarcinoma (adenocarcinoma in situ and minimally invasive adenocarcinoma). This study aims to investigate the performance of the deep learning approach based on high-resolution computed tomography (HRCT) images in the classification of tumor invasiveness and compare it with the performances of currently available approaches.MethodsIn this study, we used a deep learning approach based on 3D conventional networks to automatically predict the invasiveness of pulmonary nodules. A total of 901 early-stage non-small cell lung cancer patients who underwent surgical treatment at Shanghai Chest Hospital between November 2015 and March 2017 were retrospectively included and randomly assigned to a training set (n=814) or testing set 1 (n=87). We subsequently included 116 patients who underwent surgical treatment and intraoperative frozen section between April 2019 and January 2020 to form testing set 2. We compared the performance of our deep learning approach in predicting tumor invasiveness with that of intraoperative frozen section analysis and human experts (radiologists and surgeons).ResultsThe deep learning approach yielded an area under the receiver operating characteristic curve (AUC) of 0.946 for distinguishing preinvasive adenocarcinoma from invasive lung adenocarcinoma in the testing set 1, which is significantly higher than the AUCs of human experts (P<0.05). In testing set 2, the deep learning approach distinguished invasive adenocarcinoma from preinvasive adenocarcinoma with an AUC of 0.862, which is higher than that of frozen section analysis (0.755, P=0.043), senior thoracic surgeons (0.720, P=0.006), radiologists (0.766, P>0.05) and junior thoracic surgeons (0.768, P>0.05).ConclusionsWe developed a deep learning model that achieved comparable performance to intraoperative frozen section analysis in determining tumor invasiveness. The proposed method may contribute to clinical decisions related to the extent of surgical resection.

show abstract

Relation classification via BERT with piecewise convolution and focal loss

Cited by 11 publications

References 27 publications

BertSRC: transformer-based semantic relation classification

BertSRC: transformer-based semantic relation classification

Event-Argument Linking in Disaster Domain

3D deep learning versus the current methods for predicting tumor invasiveness of lung adenocarcinoma based on high-resolution computed tomography images

Contact Info

Product

Resources

About