ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

Abstract

Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences. While such models can provide a powerful score for matching and subsequent zero-shot tasks, they are not capable of generating caption given an image. In this work, we repurpose such models to generate a descriptive text given an image at inference time, without any further training or tuning step. This is done by combining the visual-semantic model with a large language model, benefiting from the knowledge in both web-scale models. The resulting captions are much less restrictive than those obtained by supervised captioning methods. Moreover, as a zero-shot learning method, it is extremely flexible and we demonstrate its ability to perform image arithmetic in which the inputs can be either images or text and the output is a sentence. This enables novel high-level vision capabilities such as comparing two images or solving visual analogy tests. Our code is available at: https://github. com/YoadTew/zero-shot-image-to-text.

Language Models Can See: Plugging Visual Controls in Text Generation

“…It should be emphasized that MAGIC Search allows us to directly plug visual controls into the decoding process of the language model, without the need of extra supervised training [14] or gradient update on additional features [14,78]. This property makes our method much more computationally efficient than previous approaches as demonstrated in our experiments (Section §4.1).…”

“…It has shown impressive zero-shot capabilities on various vision-language tasks and can open new avenues for answering the former question. ZeroCap [78] is the most related to our work. It is built on a pre-trained CLIP model together with the GPT-2 language model [60].…”

“…Similarly, in the text generation field, PPLM [14] extends the previous PPGN to text generation tasks (i.e., editing topic and sentiment), where the image generative models is replaced with a GPT-2 language model. Most recently, ZeroCap [78] shows image captioning task can be tackled by directly combining the existing CLIP and GPT-2 models. In general, most of these mentioned "plug and play" methods require iteratively shifting the hidden code (or latent code, or context cache) with gradient descent optimization.…”

“…Given an image, it retrieves the most related caption from the training text corpus based on the image-text similarity as measured by CLIP. (3) Lastly, we compare with the current state-of-the-art approach, ZeroCap [78], which performs CLIP-guided gradient update on the language model features during the decoding process. For a fair comparison, we use the same language model for ZeroCap as in our approach.…”

“…However, such methods are usually limited by the object detectors that are trained with a fixed set of labels. The closest to our proposal is ZeroCap [78] which is an unsupervised image captioning method by combining frozen CLIP and GPT-2. One of the advantages of ZeroCap is it performs ex post facto in the activation space without re-training or fine-tuning the CLIP and GPT-2 models.…”