Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction

Alt, Christoph; Hübner, Marc P.; Hennig, Leonhard

doi:10.18653/v1/p19-1134

Cited by 109 publications

(91 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, we investigated the necessary number of reports needed for accurate classification for our pathology reports by varying the size of the training set of reports from 16 to 256 across both classification and extraction. While others have performed sample efficiency analysis of NLP algorithms across many tasks, 28–30 to our knowledge, this has not been investigated for the important application of clinical information extraction from pathology reports, with the exception of Yala et al who plot dataset size vs performance over only one method (boosting) and over fields that only take 2 values. 6 Overall, we found that only 128 labeled reports were needed for the best methods for classification and only 64 for the token extractor, a small number compared to the dataset sizes used in prior work.…”

Section: Discussionmentioning

confidence: 99%

Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation

et al. 2020

View full text Add to dashboard Cite

Objective Cancer is a leading cause of death, but much of the diagnostic information is stored as unstructured data in pathology reports. We aim to improve uncertainty estimates of machine learning-based pathology parsers and evaluate performance in low data settings. Materials and methods Our data comes from the Urologic Outcomes Database at UCSF which includes 3232 annotated prostate cancer pathology reports from 2001 to 2018. We approach 17 separate information extraction tasks, involving a wide range of pathologic features. To handle the diverse range of fields, we required 2 statistical models, a document classification method for pathologic features with a small set of possible values and a token extraction method for pathologic features with a large set of values. For each model, we used isotonic calibration to improve the model’s estimates of its likelihood of being correct. Results Our best document classifier method, a convolutional neural network, achieves a weighted F1 score of 0.97 averaged over 12 fields and our best extraction method achieves an accuracy of 0.93 averaged over 5 fields. The performance saturates as a function of dataset size with as few as 128 data points. Furthermore, while our document classifier methods have reliable uncertainty estimates, our extraction-based methods do not, but after isotonic calibration, expected calibration error drops to below 0.03 for all extraction fields. Conclusions We find that when applying machine learning to pathology parsing, large datasets may not always be needed, and that calibration methods can improve the reliability of uncertainty estimates.

show abstract

Section: Discussionmentioning

confidence: 99%

Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation

et al. 2020

View full text Add to dashboard Cite

show abstract

“…It has also been proved that the transformer used in GPT and BERT has better ability on extracting features than the LSTM. Alt, Hübner, and Hennig (2019) have utilized a transformer decoder in GPT for distantly supervised relation extraction and obtained state-of-the-art results.…”

Section: Related Workmentioning

confidence: 99%

Joint Entity and Relation Extraction with a Hybrid Transformer and Reinforcement Learning Based Model

Xiao

Tan

Zhai

et al. 2020

AAAI

View full text Add to dashboard Cite

Joint extraction of entities and relations is a task that extracts the entity mentions and semantic relations between entities from the unstructured texts with one single model. Existing entity and relation extraction datasets usually rely on distant supervision methods which cannot identify the corresponding relations between a relation and the sentence, thus suffers from noisy labeling problem. We propose a hybrid deep neural network model to jointly extract the entities and relations, and the model is also capable of filtering noisy data. The hybrid model contains a transformer-based encoding layer, an LSTM entity detection module and a reinforcement learning-based relation classification module. The output of the transformer encoder and the entity embedding generated from the entity detection module are combined as the input state of the reinforcement learning module to improve the relation classification and noisy data filtering. We conduct experiments on the public dataset produced by the distant supervision method to verify the effectiveness of our proposed model. Different experimental results show that our model gains better performance on entity and relation extraction than the compared methods and also has the ability to filter noisy sentences.

show abstract

“…However, none of the above methods pay attention to the biased and inaccurate test set. Though human evaluation can yield accurate evaluation results (Zeng et al, 2015;Alt et al, 2019), labeling all the instances in the test set is too costly.…”

Section: Related Workmentioning

confidence: 99%

Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction

Zhang

Jia

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Distant supervision has been a widely used method for neural relation extraction for its convenience of automatically labeling datasets. However, existing works on distantly supervised relation extraction suffer from the low quality of test set, which leads to considerable biased performance evaluation. These biases not only result in unfair evaluations but also mislead the optimization of neural relation extraction. To mitigate this problem, we propose a novel evaluation method named active testing through utilizing both the noisy test set and a few manual annotations. Experiments on a widely used benchmark show that our proposed approach can yield approximately unbiased evaluations for distantly supervised relation extractors.

show abstract

Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction

Cited by 109 publications

References 33 publications

Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation

Natural language processing systems for pathology parsing in limited data environments with uncertainty estimation

Joint Entity and Relation Extraction with a Hybrid Transformer and Reinforcement Learning Based Model

Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction

Contact Info

Product

Resources

About