Boundary-sensitive Pre-training for Temporal Localization in Videos

Abstract

Video activity localization aims at understanding the semantic content in long untrimmed videos and retrieving actions of interest. The retrieved action with its start and end locations can be used for highlight generation, temporal action detection, etc. Unfortunately, learning the exact boundary location of activities is highly challenging because temporal activities are continuous in time, and there are often no clear-cut transitions between actions. Moreover, the definition of the start and end of events is subjective, which may confuse the model. To alleviate the boundary ambiguity, we propose to study the video activity localization problem from a denoising perspective. Specifically, we propose an encoder-decoder model named DenoiseLoc. During training, a set of action spans is randomly generated from the ground truth with a controlled noise scale. Then we attempt to reverse this process by boundary denoising, allowing the localizer to predict activities with precise boundaries and resulting in faster convergence speed. Experiments show that DenoiseLoc advances several video activity understanding tasks. For example, we observe a gain of +12.36% average mAP on QV-Highlights dataset and +1.64%

ActionFormer: Localizing Moments of Actions with Transformers

“…Another limitation is the need for many human labeled videos for training and the constraint of a pre-defined vocabulary of actions. Interesting future directions include pre-training for action localization [2,75], and learning from videos and text corpus [30,53] without human labels.…”

The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions

“…SNEAK [139] studies adversarial robustness of TSGV models by examining three facets of vulnerabilities, i.e., vision, language, and crossmodal interaction, from both attack and defense aspects. Xu et al [140] further investigate model pre-training for TSGV by constructing a large-scale synthesized dataset with annotations, and designing a novel boundary-sensitive pretext task. Cao et al [141] reformulate TSGV as a set prediction task, and propose a multimodal transformer model inherited from DETR [142].…”

“…However, the visual and textual features remain separately generated by different pre-trained extractors. Xu et al [140] propose a pre-training strategy for TSGV by constructing a large-scale synthesized dataset with TSGV annotations. Inspired by ViT [167], Cao et al [141] develop a video cubic embedding module to extract 3D visual tokens and learn video content from scratch without reliance on pretrained visual feature extractor.…”

LocVTP: Video-Text Pre-training for Temporal Localization

“…Compared to the retrieval tasks [64,45,21] which only require only video-level predictions, localization tasks [15,51,77] are essentially different since they need dense clip-level or frame-level predictions and thus the pre-training for these tasks is more challenging. In the pure video domain, this gap has been noticed and several pre-training works [65,2,66,73] tailored for action localization have been proposed. BSP [65] synthesizes temporal boundaries using existing action recognition datasets and conducts boundary type classification to generate localization-friendly features.…”