Frozen CLIP Models are Efficient Video Learners

Lin, Ziyi; Geng, Shijie; Zhang, Renrui; Gao, Peng; Melo, Gerard de; Wang, Xiaogang; Dai, Jifeng; Qiao, Yu; Li, Hongsheng

doi:10.1007/978-3-031-19833-5_23

Cited by 58 publications

(15 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, it is better to optimize diffusion models to enable the inverse back projection of noise sequences into a latent time-continuous video space aligned with human perception [55,32,76]. To achieve this aim, previous work incorporates the power of large language model (LLM) into the design of video generation [75], while other approaches effectively capture information using a frozen CLIP encoder [33]. However, they primarily consider semantic correlations between frames and do not adequately address dense correlations (e.g., patches, key points) across frames, which are crucial for frame continuity at a finer granularity.…”

Section: 3mentioning

confidence: 99%

Identity of university Chinese heritage language learners in Hong Kong

Li¹,

李蓁²

View full text Add to dashboard Cite

The core of out-of-distribution (OOD) detection is to learn the in-distribution (ID) representation, which is distinguishable from OOD samples. Previous work applied recognition-based methods to learn the ID features, which tend to learn shortcuts instead of comprehensive representations. In this work, we find surprisingly that simply using reconstruction-based methods could boost the performance of OOD detection significantly. We deeply explore the main contributors of OOD detection and find that reconstructionbased pretext tasks have the potential to provide a generally applicable and efficacious prior, which benefits the model in learning intrinsic data distributions of the ID dataset. Specifically, we take Masked Image Modeling as a pretext task for our OOD detection framework (MOOD). Without bells and whistles, MOOD outperforms previous SOTA of one-class OOD detection by 5.7%, multi-class OOD detection by 3.0%, and near-distribution OOD detection by 2.1%. It even defeats the 10-shot-per-class outlier exposure OOD detection, although we do not include any OOD samples for our detection.

show abstract

Section: 3mentioning

confidence: 99%

Identity of university Chinese heritage language learners in Hong Kong

Li¹,

李蓁²

View full text Add to dashboard Cite

show abstract

“…Representative works such as CLIP [43] project images and natural language descriptions to a common feature space through two separate encoders for contrastive learning, and achieve significant "zero-shot" transferability by pre-training on hundreds of millions of image-text pairs. Subsequently, these pre-trained models have been extended to various downstream tasks and shown excellent performance, including image classification [81,80], object detection [48,15], semantic segmentation [63,45], and video understanding [34,22,35]. Inspired by these successes, in this work we present the first simple but efficient framework to leverage the rich semantic knowledge of CLIP for fewshot action recognition.…”

Section: Related Workmentioning

confidence: 99%

CLIP-guided Prototype Modulating for Few-shot Action Recognition

Wang¹,

Zhang²,

Cen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Learning from large-scale contrastive languageimage pre-training like CLIP has shown remarkable success in a wide range of downstream tasks recently, but it is still under-explored on the challenging few-shot action recognition (FSAR) task. In this work, we aim to transfer the powerful multimodal knowledge of CLIP to alleviate the inaccurate prototype estimation issue due to data scarcity, which is a critical problem in low-shot regimes. To this end, we present a CLIP-guided prototype modulating framework called CLIP-FSAR, which consists of two key components: a video-text contrastive objective and a prototype modulation. Specifically, the former bridges the task discrepancy between CLIP and the few-shot video task by contrasting videos and corresponding class text descriptions. The latter leverages the transferable textual concepts from CLIP to adaptively refine visual prototypes with a temporal Transformer. By this means, CLIP-FSAR can take full advantage of the rich semantic priors in CLIP to obtain reliable prototypes and achieve accurate few-shot classification. Extensive experiments on five commonly used benchmarks demonstrate the effectiveness of our proposed method, and CLIP-FSAR significantly outperforms existing state-of-theart methods under various settings. The source code and

show abstract

“…Popular image-language models such as CLIP [83] and ALIGN [48] are trained on massive datasets by using web images and alt-text. Similarly, videolanguage models are catching up and can be categorised into two broad directions: (i) adapting image-language models for videos [8,22,49,50,62,65,71,108,110,119], and (ii) pure video-based models that are learned using large video-text datasets [3,7,[26][27][28]30,57,61,64,67,68,95,117]. Recently, a new paradigm of post-pretraining has emerged where an existing image-or video-language model goes through another stage of self-supervised pretraining on a small amount of video data before it is evaluated on downstream tasks [65,119].…”

Section: Foundational Video-language Modelsmentioning

confidence: 99%

Test of Time: Instilling Video-Language Models with a Sense of Time

Bagad¹,

Tapaswi²,

Snoek³

2023

Preprint

View full text Add to dashboard Cite

Modeling and understanding time remains a challenge in contemporary video understanding models. With language emerging as a key driver towards powerful generalization, it is imperative for foundational video-language models to have a sense of time. In this paper, we consider a specific aspect of temporal understanding: consistency of time order as elicited by before/after relations. We establish that six existing video-language models struggle to understand even such simple temporal relations. We then question whether it is feasible to equip these foundational models with temporal awareness without re-training them from scratch. Towards this, we propose a temporal adaptation recipe on top of one such model, VideoCLIP, based on post-pretraining on a small amount of video-text data. We conduct a zero-shot evaluation of the adapted models on six datasets for three downstream tasks which require a varying degree of time awareness. We observe encouraging performance gains especially when the task needs higher time awareness. Our work serves as a first step towards probing and instilling a sense of time in existing video-language models without the need for data and compute-intense training from scratch.

show abstract

Frozen CLIP Models are Efficient Video Learners

Cited by 58 publications

References 27 publications

Identity of university Chinese heritage language learners in Hong Kong

Identity of university Chinese heritage language learners in Hong Kong

CLIP-guided Prototype Modulating for Few-shot Action Recognition

Test of Time: Instilling Video-Language Models with a Sense of Time

Contact Info

Product

Resources

About