Attention on Attention for Image Captioning

Abstract

Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an "Attention on Attention" (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an "information vector" and an "attention gate" using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the "attended information", the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-ofthe-art performance of 129.8 CIDEr-D score on MS COCO "Karpathy" offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.

A Sparse Transformer-Based Approach for Image Captioning

“…The Up-Down [7] method proposed a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of salient image regions. The AoANet [9] method introduced an extension of the attention operator in which the final attended information is weighted by a gate. Our work is developed on top of these methods.…”

“…AoANet model: This model is proposed in AoANet [9] where the results of self-attention and initial query are concatenated together and fed into two linear layers to obtain information vectors by multiplication with a sigmoid gate. The final result is used as an alternative to the original selfattention operation.…”

“…Li et al [26] devise a unique attention mechanism to exploit the visual and semantic information simultaneously in the transformer framework. In addition, Huang et al [9] introduce an extension of the attention operator in which the final attended information is concatenated with the hidden state of LSTM and the attention context. For enhancing the connection between encoder and decoder, Cornia et al [10] propose a meshed transformer to learn a hierarchical representation of the relationships between image regions and use a mesh-like connectivity at the decoding stage to exploit hierarchical image features.…”

“…A captioning model normally uses two LSTM layers to produce words where the first LSTM layer is used as an attention model and the second LSTM layer as a language model shown as Figure 4(a). Followed by [9], we apply multi-head self-attention to decoder. The input vector to the first LSTM layer at each time step consists of mean pooled image feature, the previous output of the language LSTM and words:…”

“…In recent years, the great success of Transformer network [8] has attracted many researchers to explore towards the cutting edge techniques of self-attention for image captioning. The outstanding performance of selfattention comes from the ability to explore the relationships between the detected entities in encoder and model the correlation between image regions and hidden states as attention mechanism [9] [10].…”