CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims

Abstract

We introduce climate-fever, a new publicly available dataset for verification of climate change-related claims. By providing a dataset for the research community, we aim to facilitate and encourage work on improving algorithms for retrieving evidential support for climate-specific claims, addressing the underlying language understanding challenges, and ultimately help alleviate the impact of misinformation on climate change. We adapt the methodology of fever [1], the largest dataset of artificially designed claims, to reallife claims collected from the Internet. While during this process, we could rely on the expertise of renowned climate scientists, it turned out to be no easy task. We discuss the surprising, subtle complexity of modeling real-world climate-related claims within the fever framework, which we believe provides a valuable challenge for general natural language understanding. We hope that our work will mark the beginning of a new exciting long-term joint effort by the climate science and ai community.Recently, new literature on algorithmic fact-checking has emerged, using machine learning and natural language understanding (nlu) to work on this problem from different angles. One influential framework that combines several of these aspects is fever [1]. It consists of a well-vetted dataset of human-generated claims and evidence retrieved from Wikipedia and a shared-task for evaluating implementations of claim validators. Given that the fever claims are artificially constructed, they may not share the characteristics of real-world

ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction

“…et al, 2021), Touché(Bondarenko et al, 2020), Ar-guAna(Wachsmuth et al, 2018), Climate-FEVER (C-FEVER)(Diggelmann et al, 2020), FEVER(Thorne et al, 2018), Quora, SCIDOCS, and SciFact(Wadden et al, 2020).…”

Evidence-based Verification for Real World Information Needs

“…many datasets and task variants Karadzhov et al, 2017;Baly et al, 2018;Augenstein et al, 2019;Hanselowski et al, 2019;Chen et al, 2020;Wadden et al, 2020;Diggelmann et al, 2020;Schuster et al, 2021, inter alia) that have enabled the development of new models and methods for machine reading and comprehension.…”

“…The largest claim verification dataset, FEVER , contains 185,000 claims which were manually constructed; annotated with evidence supporting or refuting them from the introductory sections of Wikipedia pages. In contrast, MultiFC (Augenstein et al, 2019) (Derczynski et al, 2017) 330 yes no Twitter (Baly et al, 2018) 442 yes no News + fact checking websites (Thorne and Vlachos, 2018) 185, 445 no yes Wikipedia (constructed claims) 1, 000 yes no Debate websites (Augenstein et al, 2019) 36, 534 yes no Fact checking websites (Hanselowski et al, 2019) 6, 422 yes yes Fact checking websites (Wadden et al, 2020) 1,409 no yes Scientific articles (Diggelmann et al, 2020) 1,535 no yes 2 Wikipedia (Climate change) verification through stance classification. While stance classification considers whether claims are supported or refuted by evidence, unlike FEVER, it does not involve document retrieval or evidence extraction.…”

“…Assuming that evidence is pre-selected, similar model architectures can be applied to both settings. However, in FEVER , CLIMATE-FEVER 2 (Diggelmann et al, 2020) and SciFact (Wadden et al, 2020), individual evidence sentences must be retrieved from a corpus (such as Wikipedia). These evidence sentences were manually labelled and form part of the scoring protocol for the task.…”

Automated Fact-Checking: A Survey

“…Metadata such as page viewership statistics is helpful to rank webpages [Nie et al, 2019]. However, when search engines are not available, such as PolitiFact [Vlachos and Riedel, 2014] 106 claims Politics Very small; metadata and evidence of various forms Emergent [Ferreira and Vlachos, 2016] 300 claims News Very small; 2595 associated documents LIAR 12,836 claims Politics Medium; metadata Snopes [Popat et al, 2017] 4,956 claims Snopes website Medium; 30 Google retrieved documents for each claim FEVER [Thorne et al, 2018a] 185,445 claims Wikipedia Big; associated Wikipeida evidence LIAR-PLUS [Alhindi et al, 2018] 12,836 claims Politics Medium; automatically extracted justifications Perspectrum [Chen et al, 2019b] 907 claims Debates Small; evidence and perspectives UKP Snopes [Hanselowski et al, 2019] 6,422 claims Snopes website Medium; associated evidence MultiFC [Augenstein et al, 2019] 34,918 claims Fact-checking websites Medium; metadata and 10 Google retrieved webpages for each claim Scifact [Wadden et al, 2020] 1,409 claims Scientific papers Small; associated documents PolitiHop [Ostrowski et al, 2020] 500 claims Politics Very small; evidence chains for multi-hop reasoning WikiFactCheck-English [Sathe et al, 2020] 124,821 claims Wikipedia Big; context and evidence Climate-FEVER [Diggelmann et al, 2021] 1,535 claims Climate Medium; 7,675 claim-evidence pairs with climate related claims verified against Wikipedia evidence COVID-Fact [Saakyan et al, 2021] 4,086 claims COVID-19 Medium; 1,296 supported claims from r/COVID19 subreddit and 2,790 automatically generated refuted claims Vitamin-C [Schuster et al, 2021] 488,904 pairs Wikipedia Big; contrastive evidence from Wikipedia edits FEVEROUS [Aly et al, 2021] 87,026 claims Wikipedia Biggest; evidence collected from both structured and unstructured information on whole Wikipedia in the SCIVER shared task, the majority of effort goes into exploring similarity metrics that are used as a proxy to determine the documents' relevance to a claim. TF-IDF similarity is a common baseline [Wadden et al, 2020, Malon, 2018 and BM25 [Robertson et al, 1994] is demonstrated to be effective [Pradeep et al, 2020].…”