X-GGM: Graph Generative Modeling for Out-of-distribution Generalization in Visual Question Answering

Jiang, Jingjing; Liu, Ziyi; Liu, Yifan; Nan, Zhixiong; Zheng, Nanning

doi:10.1145/3474085.3475350

Cited by 16 publications

(7 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Beery et al (2018) [2] demonstrated a network that accurately recognizes cows in a typical context (e.g., pasture) consistently misclassifies cows in a non-typical context (e.g., beach). Similar heuristics also arise in visual question answering systems [1] and researchers proposed graph generative modeling schemes [13] (inspired by graph convolutional networks [30]) to handle the problem implicitly. In this paper, we study this problem within the Natural Language Inference (NLI): the task of determining whether a premise sentence entails (i.e., implies the truth of) a hypothesis sentence [7,8,4].…”

Section: Introductionmentioning

confidence: 90%

Understanding Out-of-distribution: A Perspective of Data Dynamics

Adila¹,

Kang²

2021

Preprint

View full text Add to dashboard Cite

Despite machine learning models' success in Natural Language Processing (NLP) tasks, predictions from these models frequently fail on out-of-distribution (OOD) samples. Prior works have focused on developing state-of-the-art methods for detecting OOD. The fundamental question of how OOD samples differ from indistribution samples remains unanswered. This paper explores how data dynamics in training models can be used to understand the fundamental differences between OOD and in-distribution samples in extensive detail. We found that syntactic characteristics of the data samples that the model consistently predicts incorrectly in both OOD and in-distribution cases directly contradict each other. In addition, we observed preliminary evidence supporting the hypothesis that models are more likely to latch on trivial syntactic heuristics (e.g., overlap of words between two sentences) when making predictions on OOD samples. We hope our preliminary study accelerates the data-centric analysis on various machine learning phenomena.

show abstract

Section: Introductionmentioning

confidence: 90%

Understanding Out-of-distribution: A Perspective of Data Dynamics

Adila¹,

Kang²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…VQA-OOD While there are certain similarities between OOD and realistic VQA, they are different. [14,31,24] address OOD where the distributions of training and test set are different. However, there is no answer for UQs, so [14,31,24] are not applicable to the proposed RVQA task.…”

Section: E Additional Related Workmentioning

confidence: 99%

“…[14,31,24] address OOD where the distributions of training and test set are different. However, there is no answer for UQs, so [14,31,24] are not applicable to the proposed RVQA task.…”

Section: E Additional Related Workmentioning

confidence: 99%

Toward Unsupervised Realistic Visual Question Answering

Zhang¹,

Ho²,

Vasconcelos³

2023

Preprint

View full text Add to dashboard Cite

The problem of realistic VQA (RVQA), where a model has to reject unanswerable questions (UQs) and answer answerable ones (AQs), is studied. We first point out 2 drawbacks in current RVQA research, where (1) datasets contain too many unchallenging UQs and (2) a large number of annotated UQs are required for training. To resolve the first drawback, we propose a new testing dataset, RGQA, which combines AQs from an existing VQA dataset with around 29K human-annotated UQs. These UQs consist of both fine-grained and coarse-grained image-question pairs generated with 2 approaches: CLIP-based and Perturbationbased. To address the second drawback, we introduce an unsupervised training approach. This combines pseudo UQs obtained by randomly pairing images and questions, with an RoI Mixup procedure to generate more fine-grained pseudo UQs, and model ensembling to regularize model confidence. Experiments show that using pseudo UQs significantly outperforms RVQA baselines. RoI Mixup and model ensembling further increase the gain. Finally, human evaluation reveals a performance gap between humans and models, showing that more RVQA research is needed.

show abstract

“…(2) Weakening language priors: AdvReg [29], GRL [30], RUBi [31], LM [32], LMH [32], Bias-Product (POE) [32], RMFE [33], CF-VQA [34] and GGE-DQ [35]; (3) Using various data enhancement: CSS [36], CL-VQA [37], GradSup [38], Loss-Rescaling [39], Mutant [40], RandImg [41], Unshuffling [42], ADA-VQA [43] and X-GGM [44].…”

Section: Language Bias In Vqamentioning

confidence: 99%

Language Bias-Driven Self-Knowledge Distillation with Generalization Uncertainty for Reducing Language Bias in Visual Question Answering

Yuan

Wang

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

To answer questions, visual question answering systems (VQA) rely on language bias but ignore the information of the images, which has negative information on its generalization. The mainstream debiased methods focus on removing language prior to inferring. However, the image samples are distributed unevenly in the dataset, so the feature sets acquired by the model often cannot cover the features (views) of the tail samples. Therefore, language bias occurs. This paper proposes a language bias-driven self-knowledge distillation framework to implicitly learn the feature sets of multi-views so as to reduce language bias. Moreover, to measure the performance of student models, the authors of this paper use a generalization uncertainty index to help student models learn unbiased visual knowledge and force them to focus more on the questions that cannot be answered based on language bias alone. In addition, the authors of this paper analyze the theory of the proposed method and verify the positive correlation between generalization uncertainty and expected test error. The authors of this paper validate the method’s effectiveness on the VQA-CP v2, VQA-CP v1 and VQA v2 datasets through extensive ablation experiments.

show abstract

X-GGM: Graph Generative Modeling for Out-of-distribution Generalization in Visual Question Answering

Cited by 16 publications

References 52 publications

Understanding Out-of-distribution: A Perspective of Data Dynamics

Understanding Out-of-distribution: A Perspective of Data Dynamics

Toward Unsupervised Realistic Visual Question Answering

Language Bias-Driven Self-Knowledge Distillation with Generalization Uncertainty for Reducing Language Bias in Visual Question Answering

Contact Info

Product

Resources

About