EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling

Abstract

While large scale pre-training has achieved great achievements in bridging the gap between vision and language, it still faces several challenges. First, the cost for pre-training is expensive. Second, there is no efficient way to handle the data noise which degrades model performance. Third, previous methods only leverage limited image-text paired data, while ignoring richer single-modal data, which may result in poor generalization to single-modal downstream tasks. In this work, we propose an EfficientCLIP method via Ensemble Confident Learning to obtain a less noisy data subset. Extra rich non-paired single-modal text data is used for boosting the generalization of text branch. We achieve the state-of-theart performance on Chinese cross-modal retrieval tasks with only 1/10 training resources compared to CLIP and WenLan, while showing excellent generalization to single-modal tasks including text retrieval and text classification.

A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision

“…[7] proposes a distillation based loss to better handle the noise in the dataset. Efficient-CLIP [44] is a concurrent work that uses text-only data to perform unimodal MLM task. DeCLIP [1] is another concurrent work that uses multiple unimodal SSL losses in addition to removing the noise from negatives with the help of nearest neighbors [11,22].…”

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

“…Vision-Language Pretraining: Joint vision-language pretraining (VLP) is an active research area [1,19,39,63,68] where the availability of large-scale image-text datasets e.g., YFCC100M [71] and Conceptual Captions [9,67] has played a key role in its progress. Although multiple concurrent works are being proposed to further improve VLP models [75], our work is different from them in a few important ways. Specifically, unlike EfficientCLIP [75] that proposes an ensemble approach to obtain a less noisy data subset for cross-modal training, our method attempts to sidestep this problem altogether by re-purposing as opposed to completely removing noisy data.…”

“…Although multiple concurrent works are being proposed to further improve VLP models [75], our work is different from them in a few important ways. Specifically, unlike EfficientCLIP [75] that proposes an ensemble approach to obtain a less noisy data subset for cross-modal training, our method attempts to sidestep this problem altogether by re-purposing as opposed to completely removing noisy data. Similarly, DeCLIP [39] improves on the data-efficiency of CLIP [63] by leveraging intra-model contrastive learning along with a nearestneighbor feature bank to augment negatives.…”

“…The convergence of self-supervised pretraining techniques in natural language processing and computer vision have brought about a renaissance of cross-modal representation learning methods [1,19,30,39,52,63,68,75] where largescale weakly correlated multimodal data (e.g., image-text pairs) is used to learn cross-modal representations using contrastive learning techniques. In particular, the recently proposed CLIP [63] model has garnered significant attention due to its impressive zero-shot recognition ability and excellent transfer performance on downstream tasks.…”

“…Specifically, we propose a simple yet effective framework for robust contrastive language-image pretraining that uses progressive self-distillation and soft image-text alignment targets to more efficiently learn from noisy data. Instead of explicitly finding, correcting or even pruning noisy correspondences [75,88], our joint student-teacher model dynamically generates a new set of soft-alignments for a random subset of images and captions in every minibatch. This enables our method to model many-to-many relationships while simultaneously re-calibrating potentially poorly matched instances without needing to identify them.…”