Counterfactual Samples Synthesizing for Robust Visual Question Answering

Abstract

Today's VQA models still tend to capture superficial linguistic correlations in the training set and fail to generalize to the test set with different QA distributions. To reduce these language biases, recent VQA works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on diagnostic benchmarks for out-of-distribution testing. However, due to complex model design, these ensemble-based methods are unable to equip themselves with two indispensable characteristics of an ideal VQA model: 1) Visual-explainable: The model should rely on the right visual regions when making decisions. 2) Question-sensitive: The model should be sensitive to the linguistic variations in questions. To this end, we propose a novel modelagnostic Counterfactual Samples Synthesizing and Training (CSST) strategy. After training with CSST, VQA models are forced to focus on all critical objects and words, which significantly improves both visual-explainable and question-sensitive abilities. Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST). CSS generates counterfactual samples by carefully masking critical objects in images or words in questions and assigning pseudo ground-truth answers. CST not only trains the VQA models with both complementary samples to predict respective ground-truth answers, but also urges the VQA models to further distinguish the original samples and superficially similar counterfactual ones. To facilitate the CST training, we propose two variants of supervised contrastive loss for VQA, and design an effective positive and negative sample selection mechanism based on CSS. Extensive experiments have shown the effectiveness of CSST. Particularly, by building on top of model LMH+SAR [1],[2], we achieve record-breaking performance on all out-of-distribution benchmarks (e.g., VQA-CP v2, v1, and GQA-OOD).

Empowering Language Understanding with Counterfactual Reasoning

“…Counterfactual sample. Constructing counterfactual samples has become an emergent data augmentation technique in natural language processing, which has been used in a wide spectral of language understanding tasks, including SA (Kaushik et al, 2019;, NLI (Kaushik et al, 2019), named entity recognition (Zeng et al, 2020) question answering (Chen et al, 2020), dialogue system , vision-language navigation (Fu et al, 2020). Beyond data augmentation under the standard supervised learning paradigm, a line of research explores to incorporate counterfactual samples into other learning paradigms such as adversarial training Fu et al, 2020;Teney et al, 2020) and contrastive learning (Liang et al, 2020).…”

“…Given the labeled factual sample, counterfactual samples can be constructed either manually(Kaushik et al, 2019) or automatically(Chen et al, 2020) by conducting minimum changes on x to swap its label from y to c…”

Deep Cross of Intra and Inter Modalities for Visual Question Answering

“…Apart from coming up with newer architecture to tackle the VQA problem, there are training techniques [18,19,20,21] put forward which might help to increase the accuracy. A special care is taken in the dataset while training which takes the semantic changes in the input data into consideration that might affect the output.…”

An Entropy Clustering Approach for Assessing Visual Question Difficulty

“…Some other works (such as DCN [39], BAN [40], and MCAN [41]) investigate "dense" co-attention that use bidirectional attention between images and questions. More recent works try to capture a more complex visual-textual information [42]- [45]. Our work instead tries to keep our approach as simple as possible by using three independently trained models to obtain the entropy.…”