Image Retrieval from Contextual Descriptions

Abstract

The ability to integrate context, including perceptual and temporal cues, plays a pivotal role in grounding the meaning of a linguistic utterance. In order to measure to what extent current vision-and-language models master this ability, we propose a new multimodal challenge, Image Retrieval from Contextual Descriptions (IMAGECODE). In particular, models are tasked with retrieving the correct image from a set of 10 minimally contrastive candidates based on a contextual description. As such, each description contains only the details that help distinguish between images. Because of this, descriptions tend to be complex in terms of syntax and discourse and require drawing pragmatic inferences. Images are sourced from both static pictures and video frames. We benchmark several state-of-the-art models, including both cross-encoders such as ViLBERT and bi-encoders such as CLIP, on IMAGECODE. Our results reveal that these models dramatically lag behind human performance: the best variant achieves an accuracy of 20.9 on video frames and 59.4 on static pictures, compared with 90.8 in humans. Furthermore, we experiment with new model variants that are better equipped to incorporate visual and temporal context into their representations, which achieve modest gains. Our hope is that IMAGECODE will foster progress in grounded language understanding by encouraging models to focus on fine-grained visual differences. We make code and dataset publicly available. 1

Communication breakdown: On the low mutual intelligibility between human and neural captioning

“…Here, we show that, when the best pre-trained image retrieval system of Krojer et al (2022) is fed captions produced by an out-of-the box neural caption generator, its performance makes a big jump forward. 0-shot image retrieval accuracy improves by almost 6% compared to the highest previously reported human-caption-based performance by the same model, with fine-tuning and various ad-hoc architectural adaptations.…”

“…Data We use the more challenging video section of the IMAGECODE data-set (Krojer et al, 2022). Since we do not fine-tune our model, we only use the validation set, including 1,872 data points.…”

“…We use the simplest CLIP-based retrieval system of Krojer et al (2022) (the one without context module and temporal embeddings), which corresponds to a standard CLIP architecture from Radford et al (2021). The caption and each image in the set are passed through a transformerbased text encoder and a transformer-based visual encoder, respectively.…”

“…Neural vision-and-language models have achieved impressive results in tasks such as visual commonsense reasoning and question answering (e.g., Chen et al, 2019;Lu et al, 2019). However, Krojer et al (2022) recently showed, in the context of captionbased image retrieval, that state-of-the-art multimodal models still perform poorly when the candidate pool contains very similar distractor images (such as close frames from the same video).…”

“…

We compare the 0-shot performance of a neural caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner. We conduct this comparison on the recently introduced IMAGECODE data-set (Krojer et al, 2022), which contains hard distractors nearly identical to the images to be retrieved. We find that the neural retriever has much higher performance when fed neural rather than human captions, despite the fact that the former, unlike the latter, were generated without awareness of the distractors that make the task hard.

…”