Evidence-based Verification for Real World Information Needs

Abstract

Claim verification is the task of predicting the veracity of written statements against evidence. Previous large-scale datasets model the task as classification, ignoring the need to retrieve evidence, or are constructed for research purposes, and may not be representative of real-world needs. In this paper, we introduce a novel claim verification dataset with instances derived from search-engine queries, yielding 10,987 claims annotated with evidence that represent real-world information needs. For each claim, we annotate evidence from full Wikipedia articles with both section and sentence-level granularity. Our annotation allows comparison between two complementary approaches to verification: stance classification, and evidence extraction followed by entailment recognition. In our comprehensive evaluation, we find no significant difference in accuracy between these two approaches. This enables systems to use evidence extraction to summarize a rationale for an end-user while maintaining the accuracy when predicting a claim's veracity. With challenging claims and evidence documents containing hundreds of sentences, our dataset presents interesting challenges that are not captured in previous work -evidenced through transfer learning experiments. We release code and data 1 to support further research on this task.

Logically at Factify 2022: Multimodal Fact Verification

“…Fact verification task requires to apply natural language inference (NLI) on long paragraphs or articles. As mentioned above, to simplify the problem, the practice of evidence sentence selection [54] that are commonly adopted in SoTA evidence-aware fact checking system are not included in our study. Thus, supported maximum sequence length and optimum document context size are two of key factors to be considered.…”