FEVER: a large-scale dataset for Fact Extraction and VERification

Thorne, James H.; Vlachos, Andreas; Christodoulopoulos, Christos; Mittal, Arpit

doi:10.48550/arxiv.1803.05355

Cited by 44 publications

(68 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…On the basis of a given sentence pair, the task is to predict 3-way labels including Support, Refute or NotEnoughInfo. Well-known shared tasks include FEVER [13] and SCIVER [14] has advanced RTE research for claim validation in recent years. This line of work performs different forms of evidence retrieval first and then perform claim validation based on that evidence.…”

Section: Related Workmentioning

confidence: 99%

Logically at Factify 2022: Multimodal Fact Verification

Gao¹,

Hoffmann²,

Oikonomou³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper describes our participant system for the multi-modal fact verification (Factify) challenge at AAAI 2022. Despite the recent advance in text based verification techniques and large pre-trained multimodal models cross vision and language, very limited work has been done in applying multimodal techniques to automate fact checking process, particularly considering the increasing prevalence of claims and fake news about images and videos on social media. In our work, the challenge is treated as multimodal entailment task and framed as multi-class classification. Two baseline approaches are proposed and explored including an ensemble model (combining two uni-modal models) and a multimodal attention network (modeling the interaction between image and text pair from claim and evidence document). We conduct several experiments investigating and benchmarking different SoTA pre-trained transformers and vision models in this work. Our best model is ranked first in leaderboard which obtains a weighted average F-measure of 0.77 on both validation and test set. Exploratory analysis of dataset is also carried out on the Factify data set and uncovers salient patterns and issues (e.g., word overlapping, visual entailment correlation, source bias) that motivates our hypothesis. Finally, we highlight challenges of the task and multimodal dataset for future research.

show abstract

Section: Related Workmentioning

confidence: 99%

Logically at Factify 2022: Multimodal Fact Verification

Gao¹,

Hoffmann²,

Oikonomou³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We focus on datasets that can be formulated as a classification problem: Movie Reviews: [63] positive/negative sentiment classification for movie reviews. FEVER: [56] a fact extraction and verification dataset where the goal is verifying claims from textual sources; each claim can either be supported or refuted. e-SNLI: [8] a natural language inference task where sentence pairs are labeled as entailment, contradiction, neutral and, supporting.…”

Section: Saliency Guided Training For Languagementioning

confidence: 99%

Improving Deep Learning Interpretability by Saliency Guided Training

Ismail¹,

Bravo²,

Feizi³

2021

Preprint

View full text Add to dashboard Cite

Saliency methods have been widely used to highlight important input features in model predictions. Most existing methods use backpropagation on a modified gradient function to generate saliency maps. Thus, noisy gradients can result in unfaithful feature attributions. In this paper, we tackle this issue and introduce a saliency guided training † procedure for neural networks to reduce noisy gradients used in predictions while retaining the predictive performance of the model. Our saliency guided training procedure iteratively masks features with small and potentially noisy gradients while maximizing the similarity of model outputs for both masked and unmasked inputs. We apply the saliency guided training procedure to various synthetic and real data sets from computer vision, natural language processing, and time series across diverse neural architectures, including Recurrent Neural Networks, Convolutional Networks, and Transformers. Through qualitative and quantitative evaluations, we show that saliency guided training procedure significantly improves model interpretability across various domains while preserving its predictive performance.

show abstract

“…Fact verification requires models to validate a claim in the context of evidence. For this task, we use the training dataset provided by the FEVER challenge [37]. The processing and split of the dataset into training/development set are conducted following Schuster et al [35] 5 .…”

Section: Experimental Settingsmentioning

confidence: 99%

Uncertainty Calibration for Ensemble-Based Debiasing Methods

Xiong¹,

Chen²,

Pang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Ensemble-based debiasing methods have been shown effective in mitigating the reliance of classifiers on specific dataset bias, by exploiting the output of a biasonly model to adjust the learning target. In this paper, we focus on the bias-only model in these ensemble-based methods, which plays an important role but has not gained much attention in the existing literature. Theoretically, we prove that the debiasing performance can be damaged by inaccurate uncertainty estimations of the bias-only model. Empirically, we show that existing bias-only models fall short in producing accurate uncertainty estimations. Motivated by these findings, we propose to conduct calibration on the bias-only model, thus achieving a three-stage ensemble-based debiasing framework, including bias modeling, model calibrating, and debiasing. Experimental results on NLI and fact verification tasks show that our proposed three-stage debiasing framework consistently outperforms the traditional two-stage one in out-of-distribution accuracy.

show abstract

FEVER: a large-scale dataset for Fact Extraction and VERification

Cited by 44 publications

References 0 publications

Logically at Factify 2022: Multimodal Fact Verification

Logically at Factify 2022: Multimodal Fact Verification

Improving Deep Learning Interpretability by Saliency Guided Training

Uncertainty Calibration for Ensemble-Based Debiasing Methods

Contact Info

Product

Resources

About