Exploring Overall Contextual Information for Image Captioning in Human-Like Cognitive Style

Abstract

Image captioning is a research hotspot where encoderdecoder models combining convolutional neural network (CNN) and long short-term memory (LSTM) achieve promising results. Despite significant progress, these models generate sentences differently from human cognitive styles. Existing models often generate a complete sentence from the first word to the end, without considering the influence of the following words on the whole sentence generation. In this paper, we explore the utilization of a human-like cognitive style, i.e., building overall cognition for the image to be described and the sentence to be constructed, for enhancing computer image understanding. This paper first proposes a Mutual-aid network structure with Bidirectional LSTMs (MaBi-LSTMs) for acquiring overall contextual information. In the training process, the forward and backward LSTMs encode the succeeding and preceding words into their respective hidden states by simultaneously constructing the whole sentence in a complementary manner. In the captioning process, the LSTM implicitly utilizes the subsequent semantic information contained in its hidden states. In fact, MaBi-LSTMs can generate two sentences in forward and backward directions. To bridge the gap between cross-domain models and generate a sentence with higher quality, we further develop a cross-modal attention mechanism to retouch the two sentences by fusing their salient parts as well as the salient areas of the image. Experimental results on the Microsoft COCO dataset show that the proposed model improves the performance of encoder-decoder models and achieves state-of-the-art results.

From Show to Tell: A Survey on Deep Learning-based Image Captioning

“…Other approaches. The solution based on additive attention over a grid of features has been widely adopted by several following works with minor improvements in terms of visual encoding [29], [32], [34], [35], [36], [37].…”

“…Hidden state reconstruction -Chen et al [34] proposed to regularize the transition dynamics of the language model by using a second LSTM for reconstructing the previous hidden state based on the current one. Ge et al [36] proposed to better capture context information by using a bidirectional LSTM with an auxiliary module. The auxiliary module in a direction approximates the hidden state of the LSTM in the other direction.…”

Self-Annotated Training for Controllable Image Captioning

“…To enhance the diversity, GAN-based methods (Dognin et al 2019;Dai et al 2017;Chen et al 2018) are introduced in image captioning. Models proposed in (Zheng, Li, and Wang 2019;Ge et al 2019) change the order of the sentence generation, starting from the middle or the end of sentences. In (Yang et al 2018;Chen et al 2020;Shi et al 2020), scene graphs are employed to further explore the objects, attributes and relationships in the image, which improve the overall performance of captioning models.…”

Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Grounding

“…Generally, existing WREG methods consist of two steps; namely, the sentence-level matching and the reconstruction [7][8][9][10]. In the first step, WREG methods roughly assume the sentence-level matching procedures from existing fullysupervised REG methods [5] in order to calculate the similarity between the entire query and each candidate proposal.…”

“…1.(a). Its accuracy, however, proves hardly satisfactory even for a full-supervised setting [9,10] and makes the BP loss unreliable. Additionally, we have the architectural imbalance from the heavy RNNstyle reconstruction network, which is never used in the final inference stage while occupying a large proportion of parameters of the entire network (around 75% in [7,8]).…”