COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

Abstract

We introduce a FEVER-like dataset COVID-Fact of 4, 086 claims concerning the COVID-19 pandemic. The dataset contains claims, evidence for the claims, and contradictory claims refuted by the evidence. Unlike previous approaches, we automatically detect true claims and their source articles and then generate counter-claims using automatic methods rather than employing human annotators. Along with our constructed resource, we formally present the task of identifying relevant evidence for the claims and verifying whether the evidence refutes or supports a given claim. In addition to scientific claims, our data contains simplified general claims from media sources, making it better suited for detecting general misinformation regarding COVID-19. Our experiments indicate that COVID-Fact will provide a challenging testbed for the development of new systems and our approach will reduce the costs of building domainspecific datasets for detecting misinformation. Original ClaimClosed environments facilitate secondary transmission of coronavirus disease 2019 Counter-Claim Closed environments prevent secondary transmission of coronavirus disease 2019 Gold Document

Synthetic Disinformation Attacks on Automated Fact Verification Systems

“…Then, these systems evaluate whether the retrieved evidence sentences validate or contradict the claim, or whether there is not enough information to make a judgment. More recently, the SCIFACT (Wadden et al 2020) and COVIDFACT (Saakyan, Chakrabarty, and Muresan 2021) benchmarks re-purposed this framework for the sci-entific domain by releasing datasets of medical claims to be verified against scientific content (Wang et al 2020). While this framework has led to impressive advances in fact verification performance (Ye et al 2020;Pradeep et al 2021), current benchmarks assume that the available evidence database contains only valid, factual information.…”

“…During document retrieval, documents in the evidence repository that are relevant to the claim are selected. Existing methods typically use information retrieval methods to rank documents based on relevance Wadden et al 2020) or use public APIs of commercial document indices (Hanselowski et al 2019;Saakyan, Chakrabarty, and Muresan 2021) to crawl related documents. In the sentence retrieval stage, individual sentences from these retrieved documents are selected with respect to their relevance to the claim, often using textual entailment (Hanselowski et al 2019), or sentence similarity methods.…”

“…This paper relies on the existing FEVER, SCIFACT, and COVIDFACT datasets, which are publicly available. To test our method, we use the same evaluation metrics proposed by the dataset authors: label accuracy for FEVER , the sentence selection, sentence label, abstract label, and abstract rationalized metrics for SCIFACT (Wadden et al 2020), and the Top-1 and Top-5 label accuracy for COVIDFACT (Saakyan, Chakrabarty, and Muresan 2021). We also introduce our own datasets of adversarial evidence generated by GROVER and PEGASUS (Zhang et al 2019).…”

The Case for Claim Difficulty Assessment in Automatic Fact Checking

“…Naturally, search based on ambiguous claims can yield poor quality search results, and thus insufficient evidence if included in a dataset to facilitate research on evidence-based fact-checking. To the best of our knowledge, this has not been considered with most existing datasets [28,1,19,23,3], though recent work on fact checking related to COVID-19 did usefully evaluate pipeline systems using Google as a baseline engine [21].…”

“…The rise of misinformation has also prompted a great body of work, especially in natural language processing (NLP), on the automatic fact checking of claims [24,20,3,11,26,10,21]. Despite tremendous progress, however, the task remains quite challenging.…”