Multimodal Video Adapter for Parameter Efficient Video Text Retrieval

Abstract

State-of-the-art video-text retrieval (VTR) methods usually fully fine-tune the pre-trained model (e.g. CLIP) on specific datasets, which may suffer from substantial storage costs in practical applications since a separate model per task needs to be stored. To overcome this issue, we present the premier work on performing parameter-efficient VTR from the pre-trained model, i.e., only a small number of parameters are tunable while freezing the backbone. Towards this goal, we propose a new method dubbed Multimodal Video Adapter (MV-Adapter) for efficiently transferring the knowledge in the pre-trained CLIP from image-text to video-text. Specifically, MV-Adapter adopts bottleneck structures in both video and text branches and introduces two novel components. The first is a Temporal Adaptation Module employed in the video branch to inject global and local temporal contexts. We also learn weights calibrations to adapt to the dynamic variations across frames. The second is a Cross-Modal Interaction Module that generates weights for video/text branches through a shared parameter space, for better aligning between modalities. Thanks to above innovations, MV-Adapter can achieve on-par or better performance than standard fine-tuning with negligible parameters overhead. Notably, on five widely used VTR benchmarks (MSR-VTT, MSVD, LSMDC, DiDemo, and ActivityNet), MV-Adapter consistently outperforms various competing methods in V2T/T2V tasks with large margins. Codes will be released.

DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval

“…Recently, (Jiang et al 2022) introduced a weight-share mechanism and adopted the query-scoring frame features reweighting method proposed in (Bain et al 2022) to boost performance. (Zhang et al 2023) proposed a temporal adaptation and cross-modal interaction modules. Prompt tuning (Lester, Al-Rfou, and Constant 2021) is another parameterefficient choice by introducing additional learnable parameters at the model's input.…”

READ-PVLA: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling

“…Previous video-language modeling methods (Liu et al 2022;Lei, Berg, and Bansal 2021;Yu et al 2021) employ pretrained Transformer models such as Unified Multimodal Transformer (UMT) (Liu et al 2022) and Vision-Guided BART (VG-BART) (Yu et al 2021), and fine-tune all the parameters of these models for every single task. This results in substantial storage overhead since each task demands storing a separate model (Zhang et al 2023). Moreover, because of the difficulty of collecting video-language data (Pan et al 2022), fully fine-tuning these over-parameterized models in low-resource scenarios, where limited training data is available, leads to instability and sub-optimal performance (Jiang et al 2022;Huang et al 2023).…”

“…To address these shortcomings, adapters are proposed as a parameter-efficient solution for finetuning video-language pretrained transformers (Jiang et al 2022;Zhang et al 2023;Yang et al 2023;Sung, Cho, and Bansal 2022;Chen et al 2022). The strategy is to add additional adaptation module to each layer of the pre-trained network and only the adaptation modules are trained during fine-tuning to improve the parameter-performance trade-off.…”