A BERT-based One-Pass Multi-Task Model for Clinical Temporal Relation Extraction

Lin, Chen; Miller, Timothy A.; Dligach, Dmitriy; Sadeque, Farig; Bethard, Steven; Savova, Guergana

doi:10.18653/v1/2020.bionlp-1.7

Cited by 35 publications

(48 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The pre-released version of BioBERT (January 2019) has already been shown to be very effective in many biomedical text mining tasks such as NER for clinical notes ( Alsentzer et al , 2019 ), human phenotype-gene RE ( Sousa et al , 2019 ) and clinical temporal RE ( Lin et al , 2019 ). The following updated versions of BioBERT will be available to the bioNLP community: (i) BioBERT and BioBERT trained on only PubMed abstracts without initialization from the existing BERT model and (ii) BioBERT and BioBERT trained on domain-specific vocabulary based on WordPiece.…”

Section: Discussionmentioning

confidence: 99%

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

et al. 2019

View full text Add to dashboard Cite

Motivation: Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. Results: We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. Availability and implementation: We make the pre-trained weights of BioBERT freely available at https://github. com/naver/biobert-pretrained, and the source code for fine-tuning

show abstract

Section: Discussionmentioning

confidence: 99%

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Previous research on extracting these relations (e.g. Bethard et al, 2017, Ning et al, 2017, Lin et al, 2019 almost always uses pair-wise TimeMLannotated data which has rich annotation but also inherits the above three complexity and consistency issues. To address these issues, Zhang and Xue (2018b) present a tree structure of relations between time expressions and events (TDT), along with a BiLSTM model (Zhang and Xue, 2018a) for parsing text into TDT and a crowd-sourced corpus (Zhang and Xue, 2019).…”

Section: Related Workmentioning

confidence: 99%

Exploring Contextualized Neural Language Models for Temporal Dependency Parsing

Ross¹,

Cai²,

Min³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Extracting temporal relations between events and time expressions has many applications such as constructing event timelines and timerelated question answering. It is a challenging problem which requires syntactic and semantic information at sentence or discourse levels, which may be captured by deep contextualized language models (LMs) such as BERT (Devlin et al., 2019). In this paper, we develop several variants of BERT-based temporal dependency parser, and show that BERT significantly improves temporal dependency parsing (Zhang and Xue, 2018a). We also present a detailed analysis on why deep contextualized neural LMs help and where they may fall short.

show abstract

“…Model Choice: recent literature has shown some attempts to use neural models to classify temporal relations in text (Lin et al, 2019). We propose to use the BERT transformer model, to solve our anchor date relation problem.…”

Section: Relation Classification Approachmentioning

confidence: 99%

“…Input Definition: While not constrained within a sentence by previous models (Lin et al, 2019), BERT was not designed to solve problems of longdistance relations within a text. There is a limitation on the size of the input text sequences it can accept (512 tokens).…”

Section: Relation Classification Approachmentioning

confidence: 99%

“…As our problem might require longer text sequences, we defined an adapted input representation. For example, the method used by Lin et al (2019) was to pass as input to BERT a single sequence of tokens with the relevant timexes tagged. For this method to work, both of the expressions from the candidate relation have to be part of a 512 token window in the text.…”

Section: Relation Classification Approachmentioning

confidence: 99%

See 1 more Smart Citation

Relative and Incomplete Time Expression Anchoring for Clinical Text

Dupuis¹,

Bergou²,

Tissot³

et al. 2020

Proceedings of the 3rd Clinical Natural Language Processing Workshop

View full text Add to dashboard Cite

Extracting and modeling temporal information in clinical text is an important element for developing timelines and disease trajectories. Time information in written text varies in preciseness and explicitness, posing challenges for NLP approaches that aim to accurately anchor temporal information on a timeline. Relative and incomplete time expressions (RI-Timexes) are expressions that require additional information for their temporal anchor to be resolved, but few studies have addressed this challenge specifically. In this study, we aimed to reproduce and verify a classification approach for identifying anchor dates and relations in clinical text, and propose a novel relation classification approach for this task.

show abstract

A BERT-based One-Pass Multi-Task Model for Clinical Temporal Relation Extraction

Cited by 35 publications

References 23 publications

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Exploring Contextualized Neural Language Models for Temporal Dependency Parsing

Relative and Incomplete Time Expression Anchoring for Clinical Text

Contact Info

Product

Resources

About