ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition

Abstract

Capitalizing on large pre-trained models for various downstream tasks of interest have recently emerged with promising performance. Due to the ever-growing model size, the standard full fine-tuning based task adaptation strategy becomes prohibitively costly in terms of model training and storage. This has led to a new research direction in parameter-efficient transfer learning. However, existing attempts typically focus on downstream tasks from the same modality (e.g., image understanding) of the pre-trained model. This creates a limit because in some specific modalities, (e.g., video understanding) such a strong pre-trained model with sufficient knowledge is less or not available. In this work, we investigate such a novel cross-modality transfer learning setting, namely parameter-efficient image-to-video transfer learning. To solve this problem, we propose a new Spatio-Temporal Adapter (ST-Adapter) for parameter-efficient fine-tuning per video task. With a built-in spatio-temporal reasoning capability in a compact design, ST-Adapter enables a pre-trained image model without temporal knowledge to reason about dynamic video content at a small (∼8%) per-task parameter cost, requiring approximately 20 times fewer updated parameters compared to previous work. Extensive experiments on video action recognition tasks show that our ST-Adapter can match or even outperform the strong full fine-tuning strategy and state-of-theart video models, whilst enjoying the advantage of parameter efficiency. * Equal contribution Preprint. Under review.

Multimodal Video Adapter for Parameter Efficient Video Text Retrieval

“…[6] proposes a simple adapter AdaptFormer based on ViTs [9]. The Convpass [23] and ST-Adapter [35] utilize the spatial invariance and the temporal information of videos respectively. Adapters are also widely used in NLP [16,18].…”

“…Afterwards, we adapt previous PETL methods including AdaptFormer [6], Convpass [23] and ST-adapter [35] to VTR as introduced in Sec. 5.1.…”

“…Some [6,16,23] are only designed for single modality (image or text) and ignore the temporal modeling and/or the interactions between multimodal features. Others bring in large parameter overhead, thus going against the purpose of PE-VTR [35]. The above analysis manifests that there still exists a large research gap in addressing PE-VTR.…”

Exploring Parameter-Efficient Fine-tuning for Improving Communication Efficiency in Federated Learning

“…Fine-tuning is a prevalent topic in centralized transfer learning, especially in this era of the "Foundation Model" (Bommasani et al, 2022). A significant line of work is to reduce the trainable parameter number, i.e., parameter-efficient fine-tuning (PEFT) (Chen et al, 2022a;Pan et al, 2022;Liu et al, 2022). This enables easier access and usage of pre-trained models by reducing the memory cost needed to conduct fine-tuning due to fewer computed gradients.…”

Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring

“…However, it is hard to get a pretrained model as powerful as CLIP in the video domain due to the unaffordable demands on computation resources and the difficulty of collecting video-text data pairs as large and diverse as image-text data. Instead of directly pursuing video-text pretrained models [17,27], a potential alternative solution that benefits video downstream tasks is to transfer the knowledge in image-text pretrained models to the video domain, which has attracted increasing attention in recent years [12,13,26,29,30,41].…”