The Fact Extraction and VERification (FEVER) Shared Task

Thorne, James; Vlachos, Andreas; Cocarascu, Oana; Christodoulopoulos, Christos; Mittal, Arpit

doi:10.18653/v1/w18-5501

Cited by 183 publications

(206 citation statements)

References 14 publications

(18 reference statements)

Supporting

Mentioning

166

Contrasting

Order By: Relevance

“…To aid with preparing their submission of 1000 instances, the organizers hosted a webbased sandbox. Breakers had access to 8 systems (4 top systems from the first FEVER shared task (Thorne et al, 2018b), the baseline from (Thorne et al, 2018a) and 3 new qualifying submissions from the 'Build-It' phase) that were hosted by the shared task organisers. Participants could experiment with attacks by submitting small samples of 50 instances for scoring twice a day via a shared task portal which returned FEVER scores of all the hosted systems.…”

Section: Task Phasesmentioning

confidence: 99%

“…The first Fact Extraction and VERification (FEVER) shared task (Thorne et al, 2018b) focused on building systems that predict whether a textual claim is SUPPORTED or REFUTED given evidence (see (Thorne et al, 2018a) for a task description), or NOTENOUGHINFORMATION in case Wikipedia does not have appropriate evidence to verify it. As automated systems for fact checking have potentially sensitive applications it is important to study the vulnerabilities of these systems, as well as the deficiencies of the datasets they are trained on.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The FEVER2.0 Shared Task

Thorne¹,

Vlachos²,

Cocarascu³

et al. 2019

Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)

Self Cite

View full text Add to dashboard Cite

We present the results of the second Fact Extraction and VERification (FEVER2.0) Shared Task. The task challenged participants to both build systems to verify factoid claims using evidence retrieved from Wikipedia and to generate adversarial attacks against other participant's systems. The shared task had three phases: building, breaking and fixing. There were 8 systems in the builder's round, three of which were new qualifying submissions for this shared task, and 5 adversaries generated instances designed to induce classification errors and one builder submitted a fixed system which had higher FEVER score and resilience than their first submission. All but one newly submitted systems attained FEVER scores higher than the best performing system from the first shared task and under adversarial evaluation, all systems exhibited losses in FEVER score. There was a great variety in adversarial attack types as well as the techniques used to generate the attacks, In this paper, we present the results of the shared task and a summary of the systems, highlighting commonalities and innovations among participating systems.

show abstract

Section: Task Phasesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The FEVER2.0 Shared Task

Thorne¹,

Vlachos²,

Cocarascu³

et al. 2019

Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the final component, we classify each of the extracted candidate evidence in terms of whether they support, refute or are just related to the claim (other). We employ the natural language inference (NLI) model from the Hexa-F system [15] (one of the best performing systems in the FEVER shared task [12]) to classify the relation between the selected evidence sentences and the claim, one of supports/refutes/other, and a similar label which expresses whether the combined set of evidence sentences supports, refutes or is simply related to the claim.…”

Section: Fact Checking Systemmentioning

confidence: 99%

Automated Fact Checking in the News Room

Miranda¹,

Nogueira²,

Mendes³

et al. 2019

The World Wide Web Conference

Self Cite

View full text Add to dashboard Cite

Fact checking is an essential task in journalism; its importance has been highlighted due to recently increased concerns and efforts in combating misinformation. In this paper, we present an automated fact checking platform which given a claim, it retrieves relevant textual evidence from a document collection, predicts whether each piece of evidence supports or refutes the claim, and returns a final verdict. We describe the architecture of the system and the user interface, focusing on the choices made to improve its user friendliness and transparency. We conduct a user study of the factchecking platform in a journalistic setting: we integrated it with a collection of news articles and provide an evaluation of the platform using feedback from journalists in their workflow. We found that the predictions of our platform were correct 58% of the time, and 59% of the returned evidence was relevant.

show abstract

“…Claim Validation Reasoning about the validity of a particular claim can be separated into three sub-tasks: document retrieval to find documents related to the claim, ED to find the relevant pieces of evidence that support or contradict the claim, and Textual Entailment (TE) to determine whether the claim follows from the evidence. The FEVER shared tasks follows this approach (Thorne et al, 2018;Thorne and Vlachos, 2019). Other approaches, such as TwoWingOS (Yin and Roth, 2018) and DeClarE (Popat et al, 2018) combine the ED and TE models into a single end-toend method.…”

Section: Related Workmentioning

confidence: 99%

Interactive Evidence Detection: train state-of-the-art model out-of-domain or simple model interactively?

Stahlhut¹

2019

Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)

View full text Add to dashboard Cite

Finding evidence is of vital importance in research as well as fact checking and an evidence detection method would be useful in speeding up this process. However, when addressing a new topic there is no training data and there are two approaches to get started. One could use large amounts of out-of-domain data to train a state-of-the-art method, or to use the small data that a person creates while working on the topic. In this paper, we address this problem in two steps. First, by simulating users who read source documents and label sentences they can use as evidence, thereby creating small amounts of training data for an interactively trained evidence detection model; and second, by comparing such an interactively trained model against a pre-trained model that has been trained on large out-of-domain data. We found that an interactively trained model not only often out-performs a state-of-the-art model but also requires significantly lower amounts of computational resources. Therefore, especially when computational resources are scarce, e.g. no GPU available, training a smaller model on the fly is preferable to training a well generalising but resource hungry out-of-domain model.

show abstract

The Fact Extraction and VERification (FEVER) Shared Task

Cited by 183 publications

References 14 publications

The FEVER2.0 Shared Task

The FEVER2.0 Shared Task

Automated Fact Checking in the News Room

Interactive Evidence Detection: train state-of-the-art model out-of-domain or simple model interactively?

Contact Info

Product

Resources

About