Zero-Shot Temporal Action Detection via Vision-Language Prompting

Abstract

Few-shot (FS) and zero-shot (ZS) learning are two different approaches for scaling temporal action detection (TAD) to new classes. The former adapts a pretrained vision model to a new task represented by as few as a single video per class, whilst the latter requires no training examples by exploiting a semantic description of the new class. In this work, we introduce a new multi-modality few-shot (MMFS) TAD problem, which can be considered as a marriage of FS-TAD and ZS-TAD by leveraging few-shot support videos and new class names jointly. To tackle this problem, we further introduce a novel MUlti-modality PromPt mETa-learning (MUPPET) method. This is enabled by efficiently bridging pretrained vision and language models whilst maximally reusing already learned capacity. Concretely, we construct multi-modal prompts by mapping support videos into the textual token space of a vision-language model using a meta-learned adapter-equipped visual semantics tokenizer. To tackle large intra-class variation, we further design a query feature regulation scheme. Extensive experiments on ActivityNetv1.3 and THUMOS14 demonstrate that our MUPPET outperforms state-of-the-art alternative methods, often by a large margin. We also show that our MUPPET can be easily extended to tackle the few-shot object detection problem and again achieves the state-ofthe-art performance on MS-COCO dataset. The code will be available in https://github.com/sauradip/ MUPPET

Adaptive Mutual Supervision for Weakly-Supervised Temporal Action Localization

“…To avoid labor and increase flexibility, some studies [30,31,32] propose learnable prompt tuning at the textual stream, showing strong low-shot generalization. In the CV domain, some recent papers [94,95,20] introduce such randomly initialized prompt tuning to handle visual tasks, e.g., image understanding [96,41,79,45] and video understanding [17,51,56]. However, these studies ignore lexical ambiguity of category names, and cases that are not easy to describe in text.…”

“…Low-Shot Temporal Action Localization considers more realistic scenarios: generalize TAL towards action categories that are unseen (zero-shot) or with several support samples (few-shot). Existing methods [20,51,88,2] most rely on foundational models pre-trained on large-scale image-caption pairs for help. Typically, E-Prompt [20] is the first to construct wide baselines with popular prompt tuning [30,31] and vanilla temporal modeling.…”

“…Typically, E-Prompt [20] is the first to construct wide baselines with popular prompt tuning [30,31] and vanilla temporal modeling. STALE [51] explores the one-stage framework to further simplify usage. Although promising, all above methods meet two main challenges: (1) For category semantics, the definition may be vague, inaccurate, or incomplete.…”

“…Splits. Following literature [20,51], we adopt two types of splits for zero-shot scenarios. The 75:25 split: train on 75% base categories and test on 25% novel categories.…”

“…In the recent literature, another line of research [20,51] considers a more challenging problem, that requires the vision system to handle both seen and unseen categories, with low-shot (zero or only few) examples at inference time, this problem is often termed as openvocabulary temporal action localization. To tackle the problem, existing studies [20,51,42] take inspiration from large-scale foundational models [59,16,80], casting the problem of action classification in the form of crossmodal retrieval, i.e., for one action videos, searching its closest category embedding in text form (e.g., "an action video of class"). However, such a design potentially suffers from the lexical ambiguities, as multiple actions may share category names, despite its differing visual appearance.…”