Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction

Li, Pengshuai; Zhang, Xinsong; Jia, Weijia; Zhao, Wei

doi:10.18653/v1/2020.findings-emnlp.20

“…( (3) GPCNNs was proposed by Li et al [9] in 2020. GPCNNs integrated gating mechanism and soft label, and adopted sentence-level attention mechanism to improve model performance.…”

Section: Experiments Resultsmentioning

confidence: 99%

“…Liu et al [8] combined the relation score represented by entity pairs with the confidence of noise labels through the joint scoring function to obtain corrected labels for specific entity pairs to improve the performance of relation extraction. Li et al [9] integrated gating mechanism and soft label, and adopted sentence-level attention mechanism to improve the performance of the model.…”

Section: Introductionmentioning

confidence: 99%

Distantly Supervised Relation Extraction Based on Residual Attention and Self Learning

Zheng,

Xu,

Liu

et al. 2024

Neural Process Lett

0

View full text Add to dashboard Cite

Relation extraction is an important task in information extraction, which aims to identify the relation between two given entities. The algorithm based on distant supervision can automatically generate a large amount of annotated data, which becomes the main method to deal with the task of relation extraction. However, previous studies rely too much on the precision of supervision information and ignore the effective supervision information hidden in the case of mislabeling, which leads to the loss of supervision information. To solve this problem, we propose the distantly supervised relation extraction model based on residual attention and self-learning. The model uses residual attention to extract features, and then uses self-learning idea to generate corrected labels for training data, which are added into the training process as supervisory signals to prevent parameter error updates caused by noisy labels. The model can not only reduce the problem of mislabeling caused by distant supervision, but also makes full use of the available supervisory information in the data to improve data utilization. Experiments show that compared with the existing mainstream baseline methods, the proposed model has higher precision and recall.

show abstract

Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction

Gao¹,

Han²,

Bai³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Distantly supervised (DS) relation extraction (RE) has attracted much attention in the past few years as it can utilize large-scale autolabeled data. However, its evaluation has long been a problem: previous works either take costly and inconsistent methods to manually examine a small sample of model predictions, or directly test models on auto-labeled datawhich, by our check, produce as much as 53% wrong labels at the entity pair level in the popular NYT10 dataset. This problem has not only led to inaccurate evaluation, but also made it hard to understand where we are and what's left to improve in the research of DS-RE. To evaluate DS-RE models more credibly, we build manually-annotated test sets for two DS-RE datasets, NYT10 and Wiki20, and thoroughly evaluate several competitive models, especially the latest pre-trained ones. The experimental results show that the manual evaluation can indicate very different conclusions from automatic ones, especially some unexpected observations, e.g., pre-trained models can achieve dominating performance while being more susceptible to false-positives compared with previous methods. We hope that both our manual test sets and observations can help advance future DS-RE research.

show abstract

Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction

Gao¹,

Han²,

Qiu³

et al. 2021

Preprint

0

View full text Add to dashboard Cite

Distantly supervised (DS) relation extraction (RE) has attracted much attention in the past few years as it can utilize large-scale autolabeled data. However, its evaluation has long been a problem: previous works either take costly and inconsistent methods to manually examine a small sample of model predictions, or directly test models on auto-labeled datawhich, by our check, produce as much as 53% wrong labels at the entity pair level in the popular NYT10 dataset. This problem has not only led to inaccurate evaluation, but also made it hard to understand where we are and what's left to improve in the research of DS-RE. To evaluate DS-RE models more credibly, we build manually-annotated test sets for two DS-RE datasets, NYT10 and Wiki20, and thoroughly evaluate several competitive models, especially the latest pre-trained ones. The experimental results show that the manual evaluation can indicate very different conclusions from automatic ones, especially some unexpected observations, e.g., pre-trained models can achieve dominating performance while being more susceptible to false-positives compared with previous methods. We hope that both our manual test sets and observations can help advance future DS-RE research.

show abstract

Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction

Cited by 3 publications

References 17 publications

Distantly Supervised Relation Extraction Based on Residual Attention and Self Learning

Distantly Supervised Relation Extraction Based on Residual Attention and Self Learning

Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction

Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction

Contact Info

Product

Resources

About