Automatic Fact-Checking with Document-level Annotations using BERT and Multiple Instance Learning

Sathe, Aalok; Park, Joonsuk

doi:10.18653/v1/2021.fever-1.11

Cited by 6 publications

(6 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reliable references have become central to Wikipedia as it acts as an educational resource in under-resourced environments [24]. Not only are Wikipedia readers affected by the quality of content, but a large set of automated fact-checking datasets and solutions rely on it as ground-truth [7,21,42,45,47], giving the online encyclopedia a central role in the AI ecosystem.…”

Section: Related Workmentioning

confidence: 99%

Longitudinal Assessment of Reference Quality on Wikipedia

Baigutanova

Myung

Sáez-Trumper

et al. 2023

Proceedings of the ACM Web Conference 2023

View full text Add to dashboard Cite

Wikipedia plays a crucial role in the integrity of the Web. This work analyzes the reliability of this global encyclopedia through the lens of its references. We operationalize the notion of reference quality by defining reference need (RN), i.e., the percentage of sentences missing a citation, and reference risk (RR), i.e., the proportion of non-authoritative references. We release Citation Detective, a tool for automatically calculating the RN score, and discover that the RN score has dropped by 20 percent point in the last decade, with more than half of verifiable statements now accompanying references. The RR score has remained below 1% over the years as a result of the efforts of the community to eliminate unreliable references. We propose pairing novice and experienced editors on the same Wikipedia article as a strategy to enhance reference quality. Our quasi-experiment indicates that such a co-editing experience can result in a lasting advantage in identifying unreliable sources in future edits. As Wikipedia is frequently used as the ground truth for numerous Web applications, our findings and suggestions on its reliability can have a far-reaching impact. We discuss the possibility of other Web services adopting Wiki-style user collaboration to eliminate unreliable content. CCS CONCEPTS• Information systems → Wikis.

show abstract

Section: Related Workmentioning

confidence: 99%

Longitudinal Assessment of Reference Quality on Wikipedia

Baigutanova

Myung

Sáez-Trumper

et al. 2023

Proceedings of the ACM Web Conference 2023

View full text Add to dashboard Cite

show abstract

“…Claim verification is typically addressed as an NLI problem (Thorne and Vlachos, 2018). Recent progress has enforced a closed-world reliance (Pratapa et al, 2020) and incorporated multiple instance learning (Sathe and Park, 2021). While data scarcity poses a major challenge on automated fact-checking , research on fewshot claim verification is limited to date.…”

Section: Claim Verificationmentioning

confidence: 99%

Active PETs: Active Data Annotation Prioritisation for Few-Shot Claim Verification with Pattern Exploiting Training

Zeng,

Zubiaga

2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

To mitigate the impact of the scarcity of labelled data on fact-checking systems, we focus on few-shot claim verification. Despite recent work on few-shot classification by proposing advanced language models, there is a dearth of research in data annotation prioritisation that improves the selection of the few shots to be labelled for optimal model performance. We propose Active PETs, a novel weighted approach that utilises an ensemble of Pattern Exploiting Training (PET) models based on various language models, to actively select unlabelled data as candidates for annotation. Using Active PETs for few-shot data selection shows consistent improvement over the baseline methods, on two technical fact-checking datasets and using six different pretrained language models. We show further improvement with Active PETs-o, which further integrates an oversampling strategy. Our approach enables effective selection of instances to be labelled where unlabelled data is abundant but resources for labelling are limited, leading to consistently improved fewshot claim verification performance. 1

show abstract

“…For datasets, various fact-checking datasets representing different real-world domains are proposed, including both naturally occurring (Augenstein et al, 2019;Gupta and Srikumar, 2021;Saakyan et al, 2021; and humancrafted (Thorne et al, 2018;Sathe et al, 2020;Schuster et al, 2021;Atanasova et al, 2022) factchecking claims. While these FV datasets focus on different domains, there is still a substantial overlap in the abilities required to verify claims across these datasets.…”

Section: Related Workmentioning

confidence: 99%

Investigating Zero- and Few-shot Generalization in Fact Verification

Pan,

Zhang,

Kan

2023

Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacifi

View full text Add to dashboard Cite

We explore zero-and few-shot generalization for fact verification (FV), which aims to generalize the FV model trained on well-resourced domains (e.g., Wikipedia) to low-resourced domains that lack human annotations. To this end, we first construct a benchmark dataset collection which contains 11 FV datasets representing 6 domains. We conduct an empirical analysis of generalization across these FV datasets, finding that current models generalize poorly. Our analysis reveals that several factors affect generalization, including dataset size, length of evidence, and the type of claims. Finally, we show that two directions of work improve generalization: 1) incorporating domain knowledge via pretraining on specialized domains, and 2) automatically generating training data via claim generation.

show abstract

Automatic Fact-Checking with Document-level Annotations using BERT and Multiple Instance Learning

Cited by 6 publications

References 30 publications

Longitudinal Assessment of Reference Quality on Wikipedia

Longitudinal Assessment of Reference Quality on Wikipedia

Active PETs: Active Data Annotation Prioritisation for Few-Shot Claim Verification with Pattern Exploiting Training

Investigating Zero- and Few-shot Generalization in Fact Verification

Contact Info

Product

Resources

About