Fast, Diverse and Accurate Image Captioning Guided by Part-Of-Speech

Abstract

Image captioning is an ambiguous problem, with many suitable captions for an image. To address ambiguity, beam search is the de facto method for sampling multiple captions. However, beam search is computationally expensive and known to produce generic captions [8,10]. To address this concern, some variational auto-encoder (VAE) [32] and generative adversarial net (GAN) [5,25] based methods have been proposed. Though diverse, GAN and VAE are less accurate. In this paper, we first predict a meaningful summary of the image, then generate the caption based on that summary. We use part-of-speech as summaries, since our summary should drive caption generation. We achieve the trifecta: (1) High accuracy for the diverse captions as evaluated by standard captioning metrics and user studies; (2) Faster computation of diverse captions compared to beam search and diverse beam search [28]; and (3) High diversity as evaluated by counting novel sentences, distinct n-grams and mutual overlap (i.e., mBleu-4) scores.

Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning

“…Our Seq-CVAE method obtains high scores on standard captioning metrics. We obtain an accuracy comparable to both the very recently proposed POS approach [10] which uses a part-ofspeech prior and also to the AG-CVAE method [40]. Both these methods use additional information in the form of object vectors from a Faster-RCNN [32] during inference.…”

“…For high-level control, one-hot encodings that represent observed objects or groups of objects are injected at the first step of the LSTM [40]. Very recently [10], more low-level control has also been discussed by conditioning on abstract representations of partof-speech tags. Again, the conditioning was achieved by changing the initial LSTM input.…”

“…However, adequately capturing the intricacies of diversity in language models is challenging. Recent works commonly resort to latent variable models augmented with more or less supervision from object detectors or part-of-speech tags [10,40]. Common to all those methods is the fact that the latent variable either only initializes the sentence generation process or is identical across the steps of generation.…”

Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network

“…To the best of our knowledge, the POS tag information of language description has not been introduced in the video captioning task. While in image captioning, Deshpande et al treated the entire POS tag sequence given by benchmark dataset as a sample, and divided them in 1024 categories by a k-medoids cluster [10], which limits the diversity of POS sequence information. He et al controlled the input of image representations based on the predefined POS tag information of each ground-truth word [16], which can hardly obtained in practical scenario.…”

“…Prior video captioning methods also neglect the syntactic structure of a sentence during the generation process. Analogic to the fact that words are the basic composition of a sentence, the part-of-speech (POS) [10] information of each word in a sentence is the basic structure of the grammar. Therefore, the POS information of the generated sentence is able to act as one prior knowledge to guide and regularize the sentence generation, if it can be obtained beforehand.…”