Object Counts! Bringing Explicit Detections Back into Image Captioning

Abstract

The use of explicit object detectors as an intermediate step to image captioning -which used to constitute an essential stage in early work -is often bypassed in the currently dominant end-to-end approaches, where the language model is conditioned directly on a midlevel image embedding. We argue that explicit detections provide rich semantic information, and can thus be used as an interpretable representation to better understand why end-to-end image captioning systems work well. We provide an in-depth analysis of end-to-end image captioning by exploring a variety of cues that can be derived from such object detections. Our study reveals that end-to-end image captioning systems rely on matching image representations to generate captions, and that encoding the frequency, size and position of objects are complementary and all play a role in forming a good image representation. It also reveals that different object categories contribute in different ways towards image captioning.

End-to-end Image Captioning Exploits Distributional Similarity in Multimodal Space

“…Although our representation is really sparse on the object interactions, it captures the basic concept of the presence of more than one object of the same kind, and thus provides extra information. A similar trend was observed by Wang et al [29], who further explored encoding the geometric and size information of objects into the representation, and by Yin and Ordonez [33], who learn interactions using a specified object-layout RNN.…”

Visually grounded generation of entailments from premises

“…Our conclusion (Section 5) is that it is feasible to frame NLI as a generation task and that incorporating non-linguistic information is potentially profitable. However, in line with recent research evaluating Vision-Language (VL) models (Shekhar et al, 2017;Wang et al, 2018;Vu et al, 2018;Tanti et al, 2019), we also find that current architectures are unable to ground textual representations in image data sufficiently.…”

“…This is especially the case since neural approaches to fundamental computer vision (CV) tasks have yielded significant improvements (LeCun et al, 2015), while also making it possible to use pretrained CV models in multimodal neural architectures, for example in tasks such as image captioning (Bernardi et al, 2016). However, recent work has cast doubt on the extent to which such models are truly exploiting image features in a multimodal space (Shekhar et al, 2017;Wang et al, 2018;Tanti et al, 2019). Indeed, Vu et al (2018) also find that image data contributes less than expected to determining the semantic relationship between premise-hypothesis pairs in the classic RTE labelling task.…”

Fast, Diverse and Accurate Image Captioning Guided by Part-Of-Speech

“…Notable advances have been made in conditioning image captioning on semantic priors of objects by using object detectors [18,30]. This conditioning is only limited to the objects (or nouns) in the caption and ignores the remainder, while our POS approach achieves coordination for the entire sentence.…”