Interpretable Visual Question Answering by Visual Grounding From Attention Supervision Mining

Zhang, Yundong; Niebles, Juan Carlos; Soto, Álvaro

doi:10.1109/wacv.2019.00043

Cited by 63 publications

(55 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Visual-Explainable Ability in VQA Models. To improve visual-explainable ability, early works [33], [34], [35] directly apply human attention as supervision to guide the models' attention maps. However, since the existence of strong biases, even with appropriate attention maps, the remaining layers of network may still disregard the visual signal [36].…”

Section: Related Workmentioning

confidence: 99%

Counterfactual Samples Synthesizing for Robust Visual Question Answering

Chen

Yan

Xiao

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

257

149

View full text Add to dashboard Cite

Today's VQA models still tend to capture superficial linguistic correlations in the training set and fail to generalize to the test set with different QA distributions. To reduce these language biases, recent VQA works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on diagnostic benchmarks for out-of-distribution testing. However, due to complex model design, these ensemble-based methods are unable to equip themselves with two indispensable characteristics of an ideal VQA model: 1) Visual-explainable: The model should rely on the right visual regions when making decisions. 2) Question-sensitive: The model should be sensitive to the linguistic variations in questions. To this end, we propose a novel modelagnostic Counterfactual Samples Synthesizing and Training (CSST) strategy. After training with CSST, VQA models are forced to focus on all critical objects and words, which significantly improves both visual-explainable and question-sensitive abilities. Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST). CSS generates counterfactual samples by carefully masking critical objects in images or words in questions and assigning pseudo ground-truth answers. CST not only trains the VQA models with both complementary samples to predict respective ground-truth answers, but also urges the VQA models to further distinguish the original samples and superficially similar counterfactual ones. To facilitate the CST training, we propose two variants of supervised contrastive loss for VQA, and design an effective positive and negative sample selection mechanism based on CSS. Extensive experiments have shown the effectiveness of CSST. Particularly, by building on top of model LMH+SAR [1],[2], we achieve record-breaking performance on all out-of-distribution benchmarks (e.g., VQA-CP v2, v1, and GQA-OOD).

show abstract

Section: Related Workmentioning

confidence: 99%

Counterfactual Samples Synthesizing for Robust Visual Question Answering

Chen

Yan

Xiao

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

257

149

View full text Add to dashboard Cite

show abstract

“…After softmax, each sentence consequently has a contextspecific distribution for each modality ET and EEG, reflecting the averaged responses of the human subjects. Following numerous other studies which have performed explicit attention supervision [34][35][36][37], we compute two attention losses as the Kullback-Leibler divergence (D KL ) from aggregate model attention weights α to ET and EEG distributions α ′′ ET and α ′ EEG . We do so for each sentence j in batches of size M for each modality to obtain eye-tracking loss L ET and EEG loss L EEG .…”

Section: Methodsmentioning

confidence: 99%

Sentiment Analysis with Cognitive Attention Supervision

McGuire

Tomuro

2021

Proceedings of the Canadian Conference on Artificial Intelligence

View full text Add to dashboard Cite

Neural network-based language models such as BERT (Bidirectional Encoder Representations from Transformers) use attention mechanisms to create contextualized representations of inputs, conceptually analogous to humans reading words in context. For the task of classifying the sentiment of texts, we ask whether BERT's attention can be informed by human cognitive data. During training, we supervise attention with eye-tracking and/or brain imaging data and combine binary sentiment classification loss with these attention losses. We find that attention supervision can be used to manipulate BERT attention to be more similar to the ground truth human data, but that there are no significant differences in sentiment classification accuracy. However, models with cognitive attention supervision more frequently misclassify different samples from the baseline models-they more often make different errors-and the errors from models with supervised attention have a higher ratio of false negatives.

show abstract

“…As fine-grained grounding becomes a potential incentive for next-generation visionlanguage systems, to what degree it can benefit remains an open question. On one hand, for VQA [4,40] the authors point out that the attention model does not attend to same regions as humans and adding attention supervision barely helps the performance. On the other hand, adding supervision to feature map attention [15,38] was found to be beneficial.…”

Section: Related Workmentioning

confidence: 99%

Grounded Video Description

Zhou

Kalantidis

Chen

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

172

155

View full text Add to dashboard Cite

Video description is one of the most challenging problems in vision and language understanding due to the large variability both on the video and language side. Models, hence, typically shortcut the difficulty in recognition and generate plausible sentences that are based on priors but are not necessarily grounded in the video. In this work, we explicitly link the sentence to the evidence in the video by annotating each noun phrase in a sentence with the corresponding bounding box in one of the frames of a video. Our dataset, ActivityNet-Entities, augments the challenging ActivityNet Captions dataset with 158k bounding box annotations, each grounding a noun phrase. This allows training video description models with this data, and importantly, evaluate how grounded or "true" such model are to the video they describe. To generate grounded captions, we propose a novel video description model which is able to exploit these bounding box annotations. We demonstrate the effectiveness of our model on our dataset, but also show how it can be applied to image description on the Flickr30k Entities dataset. We achieve state-of-the-art performance on video description, video paragraph description, and image description and demonstrate our generated sentences are better grounded in the video.

show abstract

Interpretable Visual Question Answering by Visual Grounding From Attention Supervision Mining

Cited by 63 publications

References 24 publications

Counterfactual Samples Synthesizing for Robust Visual Question Answering

Counterfactual Samples Synthesizing for Robust Visual Question Answering

Sentiment Analysis with Cognitive Attention Supervision

Grounded Video Description

Contact Info

Product

Resources

About