Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

Cui, Yufeng; Zhao, Li-Chen; Li, Feng; Li, Yangguang; Shao, Jun

doi:10.48550/arxiv.2203.05796

Cited by 4 publications

(5 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Inter-Modality Contrastive Learning: The contrastive learning is widely used for intermodality relation modelling, such as the CLIP [77] and its following-up works [19,104,[134][135][136][137][138]. The representative work SCALE [104] is trained with Self-harmonized Inter-Modality Contrastive Learning (SIMCL), which can be written as:…”

Section: Modality Interactive Learningmentioning

confidence: 99%

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

Wang¹,

Chen²,

Qian³

et al. 2023

Preprint

View full text Add to dashboard Cite

With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as BERT, ViT, GPT, etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cuttingedge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pretraining models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: https://github.com/wangxiao5791509/MultiModal BigModels Survey.

show abstract

Section: Modality Interactive Learningmentioning

confidence: 99%

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

Wang¹,

Chen²,

Qian³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Pretraining Datasets To make a fair comparison with the state-of-the-art contrastive vision-language pretraining approaches, we adopt the YFCC15M benchmark proposed in (Cui et al, 2022) which builds on a subset from YFCC100M (Thomee et al, 2016) consisting of 15M image-text pairs. In addition, we construct a 30M version of pretraining data by including Conceptual Caption 3M (CC3M) (Sharma et al, 2018) and 12M (CC12M) (Changpinyo et al, 2021).…”

Section: Experimental Settingsmentioning

confidence: 99%

“…We first compare HiCLIP with state-of-the-art CLIP family approaches on YFCC15M benchmark (Cui et al, 2022) 2022) apply multiple single-modal self-supervised tasks in addition to CLIP, we incorporated the same objectives into our hierarchy-aware model for a fair comparison (denoted as HiDeCLIP). By combining the contrastive learning and self-supervised learning loss functions, our HiDeCLIP further improves the zero-shot ImageNet classification performance by 2.7% over DeCLIP, and overall 13.1% higher than CLIP.…”

Section: Visual Recognitionmentioning

confidence: 99%

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Geng¹,

Yuan²,

Tian³

et al. 2023

Preprint

View full text Add to dashboard Cite

The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other visionlanguage models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning. To this end, we equip both the visual and language branches in CLIP with hierarchy-aware attentions, namely Hierarchy-aware CLIP (HiCLIP), to progressively discover semantic hierarchies layer-by-layer from both images and texts in an unsupervised manner. As a result, such hierarchical aggregation significantly improves the cross-modal alignment. To demonstrate the advantages of HiCLIP, we conduct qualitative analysis on its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks. 1 * This work was conducted while interning at ByteDance. 1 We release our implementation of HiCLIP at https://github.com/jeykigung/HiCLIP.

show abstract

“…To gain efficiency, cross-modality parameter sharing [52] has been proposed. Moreover, some approaches go towards the direction of fully exploiting the noisy training data by modifying the training objective with self-supervision [30], within-modality loss terms [9,23],…”

Section: Related Workmentioning

confidence: 99%

The Unreasonable Effectiveness of CLIP Features for Image Captioning: An Experimental Analysis

Barraco

Cornia

Cascianelli

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

Generating textual descriptions from visual inputs is a fundamental step towards machine intelligence, as it entails modeling the connections between the visual and textual modalities. For years, image captioning models have relied on pre-trained visual encoders and object detectors, trained on relatively small sets of data. Recently, it has been observed that large-scale multi-modal approaches like CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, provide a strong zero-shot capability on various vision tasks. In this paper, we study the advantage brought by CLIP in image captioning, employing it as a visual encoder. Through extensive experiments, we show how CLIP can significantly outperform widely-used visual encoders and quantify its role under different architectures, variants, and evaluation protocols, ranging from classical captioning performance to zero-shot transfer.

show abstract

Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

Cited by 4 publications

References 25 publications

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

The Unreasonable Effectiveness of CLIP Features for Image Captioning: An Experimental Analysis

Contact Info

Product

Resources

About