Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Huang, Zhicheng; Zeng, Zhaoyang; Huang, Yupan; Liu, Bei; Fu, Dongmei; Fu, Jianlong

doi:10.1109/cvpr46437.2021.01278

Cited by 180 publications

(128 citation statements)

References 23 publications

Supporting

Mentioning

108

Contrasting

Order By: Relevance

“…Vision-Language Pretraining. Recent years have witnessed great success in Vision-Language pretraining models [4,11,12,14,18,22,26,31,34,41,42] based on Transformer architectures [32]. Image structures have been proven useful to pretraining models, such as scene graphs [38].…”

Section: Related Workmentioning

confidence: 99%

CLIP-Event: Connecting Text and Images with Event Structures

Li¹,

Xu²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Vision-language (V+L) pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text. While existing vision-language pretraining models primarily focus on understanding objects in images or entities in text, they often ignore the alignment at the level of events and their argument structures. In this work, we propose a contrastive learning framework to enforce vision-language pretraining models to comprehend events and associated argument (participant) roles. To achieve this, we take advantage of text information extraction technologies to obtain event structural knowledge, and utilize multiple prompt functions to contrast difficult negative descriptions by manipulating event structures. We also design an event graph alignment loss based on optimal transport to capture event argument structures. In addition, we collect a large event-rich dataset (106,875 images) for pretraining, which provides a more challenging image retrieval benchmark to assess the understanding of complicated lengthy sentences 1 . Experiments show that our zero-shot CLIP-Event outperforms the state-of-the-art supervised model in argument extraction on Multimedia Event Extraction, achieving more than 5% absolute F-score gain in event extraction, as well as significant improvements on a variety of downstream tasks under zero-shot settings.

show abstract

Section: Related Workmentioning

confidence: 99%

CLIP-Event: Connecting Text and Images with Event Structures

Li¹,

Xu²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Vision-Language Pretraining: Joint vision-language pretraining has been an effective approach to improve performance on VL tasks. Many works, such as LX-MERT [45], VL-BERT [43], UNITER [10], VILLA [15], and others [19,20,25,28,52,57] have leveraged this approach. All of these works utilized datasets on the order of 10M samples for various pretraining objectives.…”

Section: Related Workmentioning

confidence: 99%

“…Considering this, most recent works that have achieved state-of-art on these tasks tend to capture logic and prior knowledge with various cross-modal transformer architectures that jointly model both modalities in a unified architecture [10,15,19,20,43,45]. These models are typically pretrained on large vision-language datasets composed of paired images and text, and optimized via a combination of supervised and self-supervised objective functions.…”

Section: Introductionmentioning

confidence: 99%

CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks

Wang¹,

Codella²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Contrastive language-image pretraining (CLIP) links vision and language modalities into a unified embedding space, yielding tremendous potential for vision-language (VL) tasks. While early concurrent works have begun to study this potential on a subset of tasks, important questions remain: 1) What is the benefit of CLIP on unstudied VL tasks? 2) Does CLIP provide benefit in low-shot or domain shifted scenarios? 3) Can CLIP improve existing approaches without impacting inference or pretraining complexity? In this work, we seek to answer these questions through two key contributions. First, we introduce an evaluation protocol that includes Visual Commonsense Reasoning (VCR), Visual Entailment (SNLI-VE), and Visual Question Answering (VQA), across a variety of data availability constraints and conditions of domain shift. Second, we propose an approach, named CLIP Targeted Distillation (CLIP-TD), to intelligently distill knowledge from CLIP into existing architectures using a dynamically weighted objective applied to adaptively selected tokens per instance. Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51.9%) and domain-shifted (up to 71.3%) conditions of VCR, while simultaneously improving performance under standard fullysupervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only. On SNLI-VE, CLIP-TD produces significant gains in low-shot conditions (up to 6.6%) as well as fully supervised (up to 3%). On VQA, CLIP-TD provides improvement in low-shot (up to 9%), and in fully-supervised (up to 1.3%). Finally, CLIP-TD outperforms concurrent works utilizing CLIP for finetuning, as well as baseline naive distillation approaches. Code will be made available.

show abstract

“…However, such kind of strategy suffers from limitations such as i) extracting region features using an object detector is computationally inefficient and ii) the quality of visual features is largely limited by the predefined visual vocabulary in pre-trained object detectors. To address this issue, rather than rely on region-based visual features, SOHO [18] takes a whole image as input and extracts compact image features through a visual dictionary, which favors 10 times faster inference time than region-based methods. ViLT [20] totally discards convolutional visual features and adopt vision transformer [11] to model long-range dependencies over a sequence of fixedsize non-overlapping image patches.…”

Section: Related Workmentioning

confidence: 99%

Vision-Language Pre-Training with Triple Contrastive Learning

Yang¹,

Duan²,

Tran³

et al. 2022

Preprint

View full text Add to dashboard Cite

Vision-language representation learning largely benefits from image-text alignment through contrastive losses (e.g., InfoNCE loss). The success of this alignment strategy is attributed to its capability in maximizing the mutual information (MI) between an image and its matched text. However, simply performing cross-modal alignment (CMA) ignores data potential within each modality, which may result in degraded representations. For instance, although CMA-based models are able to map image-text pairs close together in the embedding space, they fail to ensure that similar inputs from the same modality stay close by. This problem can get even worse when the pre-training data is noisy. In this paper, we propose triple contrastive learning (TCL) for vision-language pre-training by leveraging both cross-modal and intra-modal self-supervision. Besides CMA, TCL introduces an intra-modal contrastive objective to provide complementary benefits in representation learning. To take advantage of localized and structural information from image and text input, TCL further maximizes the average MI between local regions of image/text and their global summary. To the best of our knowledge, ours is the first work that takes into account local structure information for multi-modality representation learning. Experimental evaluations show that our approach is competitive and achieve the new state of the art on various common downstream vision-language tasks such as image-text retrieval and visual question answering.

show abstract

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Cited by 180 publications

References 23 publications

CLIP-Event: Connecting Text and Images with Event Structures

CLIP-Event: Connecting Text and Images with Event Structures

CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks

Vision-Language Pre-Training with Triple Contrastive Learning

Contact Info

Product

Resources

About