Learn and Tell: Learning Priors for Image Caption Generation

Abstract

In this work, we propose a novel priors-based attention neural network (PANN) for image captioning, which aims at incorporating two kinds of priors, i.e., the probabilities being mentioned for local region proposals (PBM priors) and part-of-speech clues for caption words (POS priors), into a visual information extraction process at each word prediction. This work was inspired by the intuitions that region proposals have different inherent probabilities for image captioning, and that the POS clues bridge the word class (part-of-speech tag) with the categories of visual features. We propose new methods to extract these two priors, in which the PBM priors are obtained by computing the similarities between the caption feature vector and local feature vectors, while the POS priors are predicated at each step of word generation by taking the hidden state of the decoder as input. After that, these two kinds of priors are further incorporated into the PANN module of the decoder to help the decoder extract more accurate visual information for the current word generation. In our experiments, we qualitatively analyzed the proposed approach and quantitatively evaluated several captioning schemes with our PANN on the MS-COCO dataset. Experimental results demonstrate that our proposed method could achieve better performance as well as the effectiveness of the proposed network for image captioning.

Research on image content description in Chinese based on fusion of image global and local features

“…They do not integrate the global features and the local features, which limits the improvement of the model description performance. Secondly, in the descriptive statement generation stage of the existing models [7,[13][14][15], the word vector with the highest probability among the candidate words is chosen by the model as the final word and output it directly as a sentence. However, the sentence obtained in this way is not necessarily the best descriptive result.…”