From captions to visual concepts and back

Abstract

This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our heldout test set, the system captions have equal or better quality 34% of the time.

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

“…Other methods map both sentences and their images to a common vector space (Ordonez et al 2011) or map them to a space of triples (Farhadi et al 2010). Among those in the second category, a common theme has been to use recurrent neural networks to produce novel captions (Kiros et al 2014;Mao et al 2014;Karpathy and Fei-Fei 2015;Vinyals et al 2015;Chen and Lawrence Zitnick 2015;Donahue et al 2015;Fang et al 2015). More recently, researchers have also used a visual attention model .…”

Movie Description

“…Recently image description has gained increased attention with work such as that of , Donahue et al (2015), Fang et al (2015), Karpathy and FeiFei (2015), Kiros et al (2014Kiros et al ( , 2015, Mao et al (2015), Vinyals et al (2015) and Xu et al (2015a). Much of the recent work has relied on Recurrent Neural Networks (RNNs) and in particular on long short-term memory networks (LSTMs).…”

A hierarchical and regional deep learning architecture for image description generation

“…Fang et al [16] proposed a caption generation system utilizing a bag of words method. Their work implements multiple instance learning and uses visual classifiers for words that commonly appear in existing captions.…”