Segmentation Guided Attention Networks for Visual Question Answering

Abstract

In this paper we propose to solve the problem of Visual Question Answering by using a novel segmentation guided attention based network which we call SegAttendNet. We use image segmentation maps, generated by a Fully Convolutional Deep Neural Network to refine our attention maps and use these refined attention maps to make the model focus on the relevant parts of the image to answer a question. The refined attention maps are used by the LSTM network to learn to produce the answer. We presently train our model on the visual7W dataset and do a category wise evaluation of the 7 question categories. We achieve state of the art results on this dataset and beat the previous benchmark on this dataset by a 1.5% margin improving the question answering accuracy from 54.1% to 55.6% and demonstrate improvements in each of the question categories. We also visualize our generated attention maps and note their improvement over the attention maps generated by the previous best approach.

Adversarial machine learning :

“…Recently, the use of ML models trained on multimodal data has gained traction, particularly the combination of image and text data modalities. Several papers have shown that multimodal models may provide some resilience against attacks [328], but other papers show that multimodal models themselves could be vulnerable to attacks mounted on all modalities at the same time [63,261,326]. See Section 4.6 for additional discussion.…”

“…Without such an effort, single modality attacks can be effective and compromise multimodal models across a wide range of multimodal tasks despite the information contained in the remaining unperturbed modalities [328,335]. Moreover, researchers have devised effcient mechanisms for constructing simultaneous attacks on multiple modalities, which suggests that multimodal models might not be more robust against adversarial attacks despite improved performance [63,261,326].…”

“…In addition, multimodal ML has made exciting progress in many tasks, including generative and classifcation tasks, and there have been attempts to use multimodal learning as a potential mitigation of single-modality attacks [328]. However, powerful simultaneous attacks against all modalities in a multimodal model have also emerged [63,261,326].…”

Attacking Distance-aware Attack: A Semi-targeted Poisoning Attack on Federated Learning

“…The field of multi-modal learning had lots of progress [2,16,19,18,17] in recent years for cross-modal understanding. The line of work in Trojan attacks on multi-modal models [20,5,27] is usually limited to a single modality, investigating the robustness to single modality Trojan and how the presence of such a Trojan would affect the multimodal model performance. For instance, Attend and Attack [20] generates adversarial visual inputs to fool a visual question answering (VQA) model [30,31,3,7,21] through a compromised attention map.…”

“…The line of work in Trojan attacks on multi-modal models [20,5,27] is usually limited to a single modality, investigating the robustness to single modality Trojan and how the presence of such a Trojan would affect the multimodal model performance. For instance, Attend and Attack [20] generates adversarial visual inputs to fool a visual question answering (VQA) model [30,31,3,7,21] through a compromised attention map. Chaturvedi et al [5] presented a targeted adversarial attack on VQA using adversarial background noise in the vision input.…”