Neural Baby Talk

Lü, Jing; Yang, Jianwei; Batra, Dhruv; Parikh, Devi

doi:10.1109/cvpr.2018.00754

Cited by 434 publications

(372 citation statements)

References 45 publications

Supporting

Mentioning

370

Contrasting

Order By: Relevance

“…Other work explores generalization to unseen combinations of visual concepts as a classification task (Misra et al, 2017;Kato et al, 2018). Lu et al (2018) is more closely related to our work; they evaluate captioning models on describing images with unseen noun-noun pairs.…”

Section: Compositional Models Of Languagementioning

confidence: 99%

Compositional Generalization in Image Captioning

Nikolaus

Abdou

Lamm

et al. 2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

Image captioning models are usually evaluated on their ability to describe a held-out set of images, not on their ability to generalize to unseen concepts. We study the problem of compositional generalization, which measures how well a model composes unseen combinations of concepts when describing images. Stateof-the-art image captioning models show poor generalization performance on this task. We propose a multi-task model to address the poor performance, that combines caption generation and image-sentence ranking, and uses a decoding mechanism that re-ranks the captions according their similarity to the image. This model is substantially better at generalizing to unseen combinations of concepts compared to state-of-the-art captioning models.

show abstract

Section: Compositional Models Of Languagementioning

confidence: 99%

Compositional Generalization in Image Captioning

Nikolaus

Abdou

Lamm

et al. 2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

show abstract

“…Similar to [2,28,48], our RDN also utilizes the attention mechanism and follows the encoder-decoder framework. However, we explicitly study the coherence between words, which remedies the drawback of current captioning framework in modeling long-term dependency in decoder.…”

Section: Related Workmentioning

confidence: 99%

Reflective Decoding Network for Image Captioning

Ke¹,

Pei²,

Li³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

State-of-the-art image captioning methods mostly focus on improving visual features, less attention has been paid to utilizing the inherent properties of language to boost captioning performance. In this paper, we show that vocabulary coherence between words and syntactic paradigm of sentences are also important to generate high-quality image caption. Following the conventional encoder-decoder framework, we propose the Reflective Decoding Network (RDN) for image captioning, which enhances both the longsequence dependency and position perception of words in a caption decoder. Our model learns to collaboratively attend on both visual and textual features and meanwhile perceive each word's relative position in the sentence to maximize the information delivered in the generated caption. We evaluate the effectiveness of our RDN on the COCO image captioning datasets and achieve superior performance over the previous methods. Further experiments reveal that our approach is particularly advantageous for hard cases with complex scenes to describe by captions.

show abstract

“…However, existing models could only describe the objects shown in the training image-caption pairs, which hinders the generalization of these models in real-world scenarios. How to describe images with unseen objects is still a challenge for image captioning [9], [10], [11].…”

Section: Introductionmentioning

confidence: 99%

Cascaded Revision Network for Novel Object Captioning

Feng

Fan

et al. 2020

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Image captioning, a challenging task where the machine automatically describes an image by sentences, has drawn significant attention in recent years. Despite the remarkable improvements of recent approaches, however, these methods are built upon a large set of training image-sentence pairs. The expensive labor efforts hence limit the captioning model to describe the wider world. In this paper, we present a novel network structure, Cascaded Revision Network, which aims at relieving the problem by equipping the model with outof-domain knowledge. CRN first tries its best to describe an image using the existing vocabulary from in-domain knowledge. Due to the lack of out-of-domain knowledge, the caption may be inaccurate or include ambiguous words for the image with unknown (novel) objects. We propose to re-edit the primary captioning sentence by a series of cascaded operations. We introduce a perplexity predictor to find out which words are most likely to be inaccurate given the input image. Thereafter, we utilize external knowledge from a pre-trained object detection model and select more accurate words from detection results by the visual matching module. In the last step, we design a semantic matching module to ensure that the novel object is fit in the right position. By this novel cascaded captioning-revising mechanism, CRN can accurately describe images with unseen objects. We validate the proposed method with state-of-the-art performance on the held-out MSCOCO dataset as well as scale to ImageNet, demonstrating the effectiveness of this method.

show abstract

Neural Baby Talk

Cited by 434 publications

References 45 publications

Compositional Generalization in Image Captioning

Compositional Generalization in Image Captioning

Reflective Decoding Network for Image Captioning

Cascaded Revision Network for Novel Object Captioning

Contact Info

Product

Resources

About