Weakly-Supervised Video Re-Localization with Multiscale Attention Model

Abstract

Video re-localization aims to localize a sub-sequence, called target segment, in an untrimmed reference video that is similar to a given query video. In this work, we propose an attention-based model to accomplish this task in a weakly supervised setting. Namely, we derive our CNN-based model without using the annotated locations of the target segments in reference videos. Our model contains three modules. First, it employs a pre-trained C3D network for feature extraction. Second, we design an attention mechanism to extract multiscale temporal features, which are then used to estimate the similarity between the query video and a reference video. Third, a localization layer detects where the target segment is in the reference video by determining whether each frame in the reference video is consistent with the query video. The resultant CNN model is derived based on the proposed co-attention loss which discriminatively separates the target segment from the reference video. This loss maximizes the similarity between the query video and the target segment while minimizing the similarity between the target segment and the rest of the reference video. Our model can be modified to fully supervised re-localization. Our method is evaluated on a public dataset and achieves the state-of-the-art performance under both weakly supervised and fully supervised settings.

Few-Shot Action Localization without Knowing Boundaries

“…In a recently proposed line of research, video re-localization, Feng et al [9] propose to localize in a query video segments that correspond semantically to a given reference video. Huang et al [15] extends the original formulation so as to learn without using temporal boundaries information in the training set by utilizing a multiscale attention module. Besides, Yang et al [52] assume only one class in each query video and more than one support videos.…”

“…Traditional fully-supervised deep learning methods typically require large amounts of annotated data, introducing a significant proneto-ambiguity annotation workload [36,37,47,55]. For this reason, learning with scarce data (i.e., few-shot learning) has received increasing attention, in domains like object detection [8,14,28,[42][43][44], action recognition [1,2,4,13,54,59], and action localization [9,15,51,52]. Current works in this domain either learn using trimmed [1,2,4,22,58,60] or well-annotated untrimmed videos [51], or address class-agnostic localization tasks [9,15,52] -learning with both scarce data and limited annotation for both action recognition and localization is still an under-explored area.…”

“…Besides, Yang et al [52] assume only one class in each query video and more than one support videos. Different with [9,15,52], we work on multi-class video localization, focusing on not only localization but also classification, which is more challenging than single-class example-based re-localizationassuming only one action, from the same class with reference video, to be located in a given query video.…”

“…A central challenge in this field, is the difficulty in obtaining large scale, fully annotated data, where the temporal extend of the different actions are given as ground truth. To address this issue, several recent works have appeared on topics such as weakly supervised localization [26,30,34,38,40,45], few-shot action detection [49,51] and video re-localization [9,15,52].…”

Survey of Action Recognition Based on Attention Mechanism

“…Chen et al [15] proposed Spatial-temporal attention-aware (STAL) to deal with the salient features of pedestrian video in spatial and temporal dimensions, this method mainly calculates the score of spatial and temporal feature information, so as to obtain the salient area. Huang et al [16] proposed a CNN model based on coattention mechanism, which is used to estimate the similarity between video actions and extract multi-scale temporal features. Shu et al [17] proposed a new Skeleton-joint Co-Attention (SCA) mechanism on RNN, which is extracted for subsequent human action recognition from sequences by learning the skeleton joint cooperative attention feature map.…”