FEVER: a Large-scale Dataset for Fact Extraction and VERification

Abstract

In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from.The claims are classified as SUPPORTED, RE-FUTED or NOTENOUGHINFO by annotators achieving 0.6841 in Fleiss κ. For the first two classes, the annotators also recorded the sentence(s) forming the necessary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach and compare it to suitably designed oracles. The best accuracy we achieve on labeling a claim accompanied by the correct evidence is 31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources.

MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims

“…Some use additional Twitter-specific features (Enayet and El-Beltagy, 2017). More involved methods taking into account evidence documents, often trained on larger datasets, consist of evidence identification and ranking following a neural model that measures the compatibility between claim and evidence (Thorne et al, 2018;Mihaylova et al, 2018;Yin and Roth, 2018).…”

BERT for Evidence Retrieval and Claim Verification

“…In the FEVER benchmark (Thorne et al, 2018), the DrQA (Chen et al, 2017) retrieval component is considered as the baseline. They choose the k-nearest documents based on the cosine similarity of TF-IDF feature vectors.…”

“…In order to extract evidence sentences, (Thorne et al, 2018) use a TF-IDF approach similar to their document retrieval. The UCL team (Yoneda et al, 2018) trains a logistic regression model on a heuristically set of features.…”

“…Decomposable Attention (DA) (Parikh et al, 2016), which compares and aggregates softaligned words in sentence pairs, is used in the FEVER benchmark paper (Thorne et al, 2018). The Papelo team (Malon, 2018) employs transformer networks with pre-trained weights (Radford et al, 2018).…”

Contrastive Language Adaptation for Cross-Lingual Stance Detection

“…The rise of social media has enabled the phenomenon of "fake news," which could target specific individuals and can be used for deceptive purposes (Lazer et al, 2018;Vosoughi et al, 2018). As manual fact-checking is a time-consuming and tedious process, computational approaches have been proposed as a possible alternative (Popat et al, 2017;Wang, 2017;Mihaylova et al, 2018), based on information sources such as social media (Ma et al, 2017), Wikipedia (Thorne et al, 2018), and knowledge bases (Huynh and Papotti, 2018). Fact-checking is a multi-step process (Vlachos and Riedel, 2014): (i) checking the reliability of media sources, (ii) retrieving potentially relevant documents from reliable sources as evidence for each target claim, (iii) predicting the stance of each document with respect to the target claim, and finally (iv) making a decision based on the stances from (iii) for all documents from (ii).…”