SLIP: Self-supervision meets Language-Image Pre-training

Abstract

Recent work has shown that self-supervised pre-training leads to improvements over supervised learning on challenging visual recognition tasks. CLIP, an exciting new approach to learning with language supervision, demonstrates promising performance on a wide variety of benchmarks. In this work, we explore whether self-supervised learning can aid in the use of language supervision for visual representation learning. We introduce SLIP, a multi-task learning framework for combining self-supervised learning and CLIP pre-training. After pre-training with Vision Transformers, we thoroughly evaluate representation quality and compare performance to both CLIP and self-supervised learning under three distinct settings: zero-shot transfer, linear classification, and end-to-end finetuning. Across Im-ageNet and a battery of additional datasets, we find that SLIP improves accuracy by a large margin. We validate our results further with experiments on different model sizes, training schedules, and pre-training datasets. Our findings show that SLIP enjoys the best of both worlds: better performance than self-supervision (+8.1% linear accuracy) and language supervision (+5.2% zero-shot accuracy).

Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

“…Training Available Image code data encoder CLIP [18] No YFCC15M-V1 † ViT, ResNet SLIP [15] Yes YFCC15M-V1 † ViT DeCLIP [9] Yes YFCC15M-V2 ViT, ResNet FILIP [30] No -ViT els from language supervision, or more specifically, imagetext pairs. Basically, CLIP adopts the contrastive loss to push the embeddings of matched image-text pairs together while pushing those of non-matched pairs apart.…”

“…Witnessing its great success, researchers continue to push the frontier of CLIP. For instance, SLIP [15], De-CLIP [9] and FILIP [30] achieve considerable improvements via embracing different kinds of supervision within the image-text pairs. However, it remains challenging to make fair comparison between these methods.…”

“…As we can see from Tab. 1, although CLIP [18], DeCLIP [9] and SLIP [15] use the same amount of 15 million data from YFCC [22], they adopt different filtering strategies. Moreover, methods [7,9,17,18,30] crawl their datasets from the Internet, making the fair comparison more difficult.…”

“…Intuitively, fine-grained alignment needs the image features to be non-overlapped, which is unachievable for ConvNets. For ViT image encoder, aggregating self-supervision [9,15], multi-view supervision [9], nearest-neighbor supervision [9] and fine-grained alignment supervision [30] brings us the strongest variant DeFILIP.…”

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

“…This paradigm is also referred to as transfer learning. Recently, image-text pre-training has become increasingly popular in computer vision as a pre-training task [72,47,65,69]. Recent work has explored alternative strategies for adapting these models to specific target tasks [106,35,105], for instance via a lightweight residual feature adapter.…”