Scaling Up Vision-Language Pre-training for Image Captioning

Abstract

In recent years, we have witnessed significant performance boost in the image captioning task based on visionlanguage pre-training (VLP). Scale is believed to be an important factor for this advance. However, most existing work only focuses on pre-training transformers with moderate sizes (e.g., 12 or 24 layers) on roughly 4 million images. In this paper, we present LEMON , a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning. We use the state-of-the-art VinVL model as our reference model, which consists of an image feature extractor and a transformer model, and scale the transformer both up and down, with model sizes ranging from 13 to 675 million parameters. In terms of data, we conduct experiments with up to 200 million image-text pairs which are automatically collected from web based on the alt attribute of the image (dubbed as ALT200M). Extensive analysis helps to characterize the performance trend as the model size and the pre-training data size increase. We also compare different training recipes, especially for training on large-scale noisy data. As a result, LEMON achieves new state of the arts on several major image captioning benchmarks, including COCO Caption, nocaps, and Conceptual Captions. We also show LEMON can generate captions with long-tail visual concepts when used in a zero-shot manner.

Language Models Can See: Plugging Visual Controls in Text Generation

“…Table 1 shows the results on zero-shot image captioning. For a comprehensive comparison, we also include the results of several representative (1) supervised methods: BUTD [3], GVD [93], UniVLP [94], ClipCap [51], Oscar [40], and LEMON [26]; and (2) weakly supervised methods: UIC [19], IC-SME [37], S2S-SS and S2S-GCC [25].…”

“…Beyond describing the whole image scene, dense captioning methods [30,5,35,88,89] aim to describe the visual objects in a sub-region of the input image. Recently, vision-language pre-training methods [94,40,51,26], benefiting from the rich visual-textual representation of pre-trained models on large-scale datasets, are tendencies for vision-language generation by re-training or fine-tuning the model parameters on downstream tasks. Although these methods have achieved impressive results, a certain amount of paired image-text data is indispensable during training.…”

CaMEL: Mean Teacher Learning for Image Captioning

“…Despite RNN-based language models have been the standard strategy for generating the caption, convolutional language models [40] and fully-attentive language models [14], [41], [42], [43], [44] based on the Transformer paradigm [45] have been explored for image captioning, also motivated by the success of these approaches on Natural Language Processing tasks such as machine translation and language understandings [45], [46], [47]. Moreover, the introduction of Transformer-based language models has brought to the development of effective variants or modifications of the self-attention operator [7], [11], [12], [13], [48], [49], [8] and has enabled vision-and-language early-fusion [19], [22], [50], based on BERT-like architectures [46].…”

“…These are in the form of soft-labels, which the captioning model has to align with in the cross-entropy phase, and reweighting of the caption words to guide the fine-tuning phase. Additional improvement to the performance of recent selfattention-based image captioning approaches is due to the use of large-scale vision-and-language pre-training [17], [19], [20], [22], [50], which can be done on noisy image-text pairs, also exploiting pre-training losses different from cross-entropy, such as the masked token loss [19], [20]. Different from previous methods, our approach is based on the interplay of two different language models that are trained with the mean teacher learning paradigm and knowledge distillation, without relying on large-scale pre-training.…”

“…Nowadays, captioning approaches have significantly evolved towards the usage of Transformer-based language models [11], [12], [13], [14], [15], while the visual feature extraction stage is rapidly evolving towards the use of grid-like features extracted with multi-modal architectures trained on large-scale data with language supervision [16], [17], [18]. In parallel, while many previous captioning approaches have been trained on middlesize datasets like COCO, the literature is now investigating the usage of large-scale noisy datasets as well [17], [19], [20], [21], [22].…”