Combining Fact Extraction and Verification with Neural Semantic Matching Networks

Abstract

The increasing concern with misinformation has stimulated research efforts on automatic fact checking. The recentlyreleased FEVER dataset introduced a benchmark factverification task in which a system is asked to verify a claim using evidential sentences from Wikipedia documents. In this paper, we present a connected system consisting of three homogeneous neural semantic matching models that conduct document retrieval, sentence selection, and claim verification jointly for fact extraction and verification. For evidence retrieval (document retrieval and sentence selection), unlike traditional vector space IR models in which queries and sources are matched in some pre-designed term vector space, we develop neural models to perform deep semantic matching from raw textual input, assuming no intermediate term representation and no access to structured external knowledge bases. We also show that Pageview frequency can also help improve the performance of evidence retrieval results, that later can be matched by using our neural semantic matching network. For claim verification, unlike previous approaches that simply feed upstream retrieved evidence and the claim to a natural language inference (NLI) model, we further enhance the NLI model by providing it with internal semantic relatedness scores (hence integrating it with the evidence retrieval modules) and ontological WordNet features. Experiments on the FEVER dataset indicate that (1) our neural semantic matching method outperforms popular TF-IDF and encoder models, by significant margins on all evidence retrieval metrics, (2) the additional relatedness score and WordNet features improve the NLI model via better semantic awareness, and (3) by formalizing all three subtasks as a similar semantic matching problem and improving on all three stages, the complete model is able to achieve the state-of-the-art results on the FEVER test set (two times greater than baseline results). 1 2 The task is often termed as natural language inference (NLI).

Revealing the Importance of Semantic Retrieval for Machine Reading at Scale

“…The system procedure is listed below: (1) Term-Based Retrieval: To begin with, we used a combination of the TF-IDF method and a rule-based keyword matching method 2 to narrow the scope from whole Wikipedia down to a set of related paragraphs; this is a standard procedure in MRS (Chen et al, 2017;Nie et al, 2019). The focus of this step is to efficiently select a candidate set P I that can cover the information as much as possible (P I ⊂ K) while keeping the size of the set acceptable enough for downstream processing.…”

“…To the best of our knowledge, we are the first to apply and optimize neural semantic retrieval at both paragraph and sentence levels for MRS. Automatic Fact Checking: Recent work formalized the task of automatic fact checking from the viewpoint of machine learning and NLP. The release of FEVER stimulates many recent developments (Nie et al, 2019;Yoneda et al, 2018;Hanselowski et al, 2018) on data-driven neural networks for automatic fact checking. We consider the task also as MRS because they share almost the same setup except that the downstream task is verification or natural language inference (NLI) rather than QA.…”

BERT for Evidence Retrieval and Claim Verification

“…They rank pages by logistic regression and extra features like capitalization, sentence position and token matching. Keyword matching along with page-view statistics are used in (Nie et al, 2019). UKP-Athene (Hanselowski et al, 2018), the highest document retrieval scoring team, uses MediaWiki API 1 to search the Wikipedia database for the claims noun phrases.…”

“…The Papelo team (Malon, 2018) employs transformer networks with pre-trained weights (Radford et al, 2018). ESIM has been widely used among the FEVER challenge participants (Nie et al, 2019;Yoneda et al, 2018;Hanselowski et al, 2018). UNC (Nie et al, 2019), the winner of the competition, proposes a modified ESIM that takes the concatenation of the retrieved evidence sentences and claim along with ELMo embedding and three additional token-level features: Word-Net, number embedding, and semantic relatedness score from the document retrieval and sentence retrieval steps.…”

“…The UNC and DREAM precision scores are better than our methods without a decision threshold, however, a threshold can regulate the trade-off between the recall and precision, and achieve the best precision and F1 scores. As discussed in (Nie et al, 2019), the recall is the most important factor. It is because the sentence retrieval predictions are the samples that we train the verification system on.…”