Gated Hierarchical Attention for Image Captioning

Abstract

Attention modules connecting encoder and decoders have been widely applied in the field of object recognition, image captioning, visual question answering and neural machine translation, and significantly improves the performance. In this paper, we propose a bottomup gated hierarchical attention (GHA) mechanism for image captioning. Our proposed model employs a CNN as the decoder which is able to learn different concepts at different layers, and apparently, different concepts correspond to different areas of an image. Therefore, we develop the GHA in which low-level concepts are merged into high-level concepts and simultaneously low-level attended features pass to the top to make predictions. Our GHA significantly improves the performance of the model that only applies one level attention, e.g., the CIDEr score increases from 0.923 to 0.999, which is comparable to the state-of-theart models that employ attributes boosting and reinforcement learning (RL). We also conduct extensive experiments to analyze the CNN decoder and our proposed GHA, and we find that deeper decoders cannot obtain better performance, and when the convolutional decoder becomes deeper the model is likely to collapse during training. Code is available: https://github.com/qingzwang/GHA-ImageCaptioning. Keywords: Hierarchical Attention · Image Captioning · Convolutional Decoder. Recently, CNNs are the most popular vision module, such as VGG nets [33], Google nets [35] and residual nets [14] (in this paper, we call them Image-CNNs). It is believed that introducing more information benefits the performance, and hence some models employ object detection or transfer image features into attributes to obtain more details or semantic information of an image [2,9,46,42,45,11]. However, applying object detection or attributes boosting

Describing Like Humans: On Diversity in Image Captioning

“…In [19], a sentinel gate decides whether the visual feature or the semantic feature should be used for prediction. Instead of employing LSTM decoders for sentences, [2,31,32] apply convolutional decoders, which achieves faster training process and comparable results.…”

Group-based Distinctive Image Captioning with Memory Attention

“…For instance, Anderson et al [1] proposes bottom-up features, which are extracted by a pre-trained Faster-RCNN [23] and a top-down attention LSTM, where an object is attended in each step when predicting captions. Apart from using RNNs as the language decoder, Aneja et al [2], Wang and Chan [33,34] utilize CNNs since LSTMs cannot be trained in a parallel manner. Cornia et al [6], Li et al [15] adopt transformer-based networks with multi-head attention to generate captions, which mitigate the long-term dependency problem in LSTMs and significantly improves the performance of image captioning.…”

Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning

“…Earlier, Xu et al [20] have introduced a spatial attention model using image feature maps to generate image captions extended with a channel-wise attention module in [21]. Later, [22] introduces a gated hierarchical attention module by merging low-level features with high-level features. Xu et al [26] propose an attention-gated LSTM model where the output gate incorporates visual attention and forwards to the cell state of LSTM.…”