Image Captioning: Transforming Objects into Words

Simao, Herdade,; Kappeler, Armin; Boakye, Kofi; Soares, João B. P.

doi:10.48550/arxiv.1906.05963

Cited by 22 publications

(35 citation statements)

References 19 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To further leverage the visual cues, an attention mechanism is usually utilized [4,6,42] to focus on specific visual features. Moreover, recent models apply self-attention [16,43] or use an expressive visual Transformer [12] as an encoder [23]. Our work uses the expressive embedding of CLIP for visual representation.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

ClipCap: CLIP Prefix for Image Captioning

Mokady¹,

Hertz²,

Bermano³

2021

Preprint

108

137

View full text Add to dashboard Cite

Image captioning is a fundamental task in visionlanguage understanding, where the model predicts a textual informative caption to a given input image. In this paper, we present a simple approach to address this task. We use CLIP encoding as a prefix to the caption, by employing a simple mapping network, and then fine-tunes a language model to generate the image captions. The recently proposed CLIP model contains rich semantic features which were trained with textual context, making it best for vision-language perception. Our key idea is that together with a pre-trained language model (GPT2), we obtain a wide understanding of both visual and textual data. Hence, our approach only requires rather quick training to produce a competent captioning model. Without additional annotations or pre-training, it efficiently generates meaningful captions for large-scale and diverse datasets. Surprisingly, our method works well even when only the mapping network is trained, while both CLIP and the language model remain frozen, allowing a lighter architecture with less trainable parameters. Through quantitative evaluation, we demonstrate our model achieves comparable results to state-of-the-art methods on the challenging Conceptual Captions and nocaps datasets, while it is simpler, faster, and lighter. Our code is available in https://github. com/rmokady/CLIP_prefix_caption.

show abstract

Section: Related Workmentioning

confidence: 99%

“…To produce the caption itself, a textual decoder is employed. Early works have used LSTM variants [8,38,39], while recent works [16,26] adopted the improved transformer architecture [36]. Built upon the transformer, one of the most notable works is BERT [11], demonstrating the dominance of the newly introduced paradigm.…”

Section: Related Workmentioning

confidence: 99%

ClipCap: CLIP Prefix for Image Captioning

Mokady¹,

Hertz²,

Bermano³

2021

Preprint

108

137

View full text Add to dashboard Cite

show abstract

“…There is a rich literature on image captioning studying different model structures and learning approaches. Recent works have proposed enhanced attention-based models to improve the performance, such as ORT [14], AoANet [16], M 2 Transformer [8], X-LAN [37], and RSTNet [57]. Besides, researchers have explored to leverage semantic attributes [26,53], scene graphs [50], and graph convolutional networks [52] for captioning.…”

Section: A Related Work On Image Captioningmentioning

confidence: 99%

Scaling Up Vision-Language Pre-training for Image Captioning

Hu¹,

Gan²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, we have witnessed significant performance boost in the image captioning task based on visionlanguage pre-training (VLP). Scale is believed to be an important factor for this advance. However, most existing work only focuses on pre-training transformers with moderate sizes (e.g., 12 or 24 layers) on roughly 4 million images. In this paper, we present LEMON , a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning. We use the state-of-the-art VinVL model as our reference model, which consists of an image feature extractor and a transformer model, and scale the transformer both up and down, with model sizes ranging from 13 to 675 million parameters. In terms of data, we conduct experiments with up to 200 million image-text pairs which are automatically collected from web based on the alt attribute of the image (dubbed as ALT200M). Extensive analysis helps to characterize the performance trend as the model size and the pre-training data size increase. We also compare different training recipes, especially for training on large-scale noisy data. As a result, LEMON achieves new state of the arts on several major image captioning benchmarks, including COCO Caption, nocaps, and Conceptual Captions. We also show LEMON can generate captions with long-tail visual concepts when used in a zero-shot manner.

show abstract

“…Prior captioning approaches have involved attention mechanisms and their variants to capture spatial relationship between objects [6] for generating captions. In this paper, we explore both object detection (image tagging) and captioning as features for assisting the ad text-to-image query task.…”

Section: Textual Description Of Images: Object Detection and Image Ca...mentioning

confidence: 99%

“…For each image the model returns a list of inferred tags with confidence scores (we used tags with confidence above 0.8). For captioning, we used an object relation transformer (ORT) model [6] with a Faster R-CNN object detector. For our analysis, we trained two ORT models: one on Microsoft COCO 2014 Captions dataset [13] (COCO-captions model), and the other on Conceptual Captions dataset [21] (CC-captions model).…”

Section: Image Tags and Caption Metadatamentioning

confidence: 99%

VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search

Mishra

Kuznetsov

Srivastava

et al. 2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Numerous online stock image libraries offer high quality yet copyright free images for use in marketing campaigns. To assist advertisers in navigating such third party libraries, we study the problem of automatically fetching relevant ad images given the ad text (via a short textual query for images). Motivated by our observations in logged data on ad image search queries (given ad text), we formulate a keyword extraction problem, where a keyword extracted from the ad text (or its augmented version) serves as the ad image query. In this context, we propose VisualTextRank: an unsupervised method to (i) augment input ad text using semantically similar ads, and (ii) extract the image query from the augmented ad text. Visu-alTextRank builds on prior work on graph based context extraction (biased TextRank in particular) by leveraging both the text and image of similar ads for better keyword extraction, and using advertiser category specific biasing with sentence-BERT embeddings. Using data collected from the Verizon Media Native (Yahoo Gemini) ad platform's stock image search feature for onboarding advertisers, we demonstrate the superiority of VisualTextRank compared to competitive keyword extraction baselines (including an 11% accuracy lift over biased TextRank). For the case when the stock image library is restricted to English queries, we show the effectiveness of VisualTextRank on multilingual ads (translated to English) while leveraging semantically similar English ads. Online tests with a simplified version of VisualTextRank led to a 28.7% increase in the usage of stock image search, and a 41.6% increase in the advertiser onboarding rate in the Verizon Media Native ad platform. CCS CONCEPTS• Information systems → Online advertising.

show abstract

Image Captioning: Transforming Objects into Words

Cited by 22 publications

References 19 publications

ClipCap: CLIP Prefix for Image Captioning

ClipCap: CLIP Prefix for Image Captioning

Scaling Up Vision-Language Pre-training for Image Captioning

VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search

Contact Info

Product

Resources

About