Stacked Attention Networks for Image Question Answering

Abstract

This paper presents stacked attention networks (SANs) that learn to answer natural language questions from images. SANs use semantic representation of a question as query to search for the regions in an image that are related to the answer. We argue that image question answering (QA) often requires multiple steps of reasoning. Thus, we develop a multiple-layer SAN in which we query an image multiple times to infer the answer progressively. Experiments conducted on four image QA data sets demonstrate that the proposed SANs significantly outperform previous state-of-the-art approaches. The visualization of the attention layers illustrates the progress that the SAN locates the relevant visual clues that lead to the answer of the question layer-by-layer.

Multimodal Cross-guided Attention Networks for Visual Question Answering

“…The second two rows of Table 1 exhibit the performance of methods with one-glimpse attention mechanism. SAN [4] employs element-wise addition operation to calculate attention map and gets better performance than the methods in the first part. In MCAN, we replace the addition or the multiplication operation with concatenated operation to compute the joint learning attention map, and gain further improvements.…”

“…A number of methods employ question-guided attention to solve VQA task. Yang et al [4] introduced the soft attention, and proposed a stacked attention model which used question representations to query question-related regions in image via multi-step reasoning. Noh et al [15] adopted visual attention with joint loss minimization.…”

“…Many previous methods [1][2][3]7] adopt this approach, while some [5,8] solve VQA task by modifying the basic idea. Besides LSTM, these approaches [3, 9-11] adopted GRU to extract high-level semantic and some [4,12,13] utilized CNN to encode question. There are several methods different from above ones, which addressed VQA task as a multi-way classification problem.…”

“…However, these methods can't locate the visual finegrained regions related to the question. Therefore, many approaches [4,5] picked up spatial representation from CNN and introduced visual attention mechanism to attend to the finegrained regions relevant to the question. The improvements are significant.…”

Mean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task

“…Our approach in the combination with the Normalized Correlation Analysis embedding technique improves on the state-of-the-art of the Visual Madlibs task. Text-Embedding Loss: Motivated by the popularity of deep architectures for visual question answering, that combine a global CNN image representation with an LSTM [7] question representation [4,13,17,20,29,30,31], as well as the leading performance of nCCA on the multi-choice Visual Madlibs task [32], we propose a novel extension of the CNN+LSTM architecture that chooses a prompt completion out of four candidates (see Figure 4) by measuring similarities directly in the embedding space. This contrasts with the prior approach of [32] that uses a post-hoc comparison between the discrete output of the CNN+LSTM method and all four candidates.…”