Weakly Supervised Action Localization by Sparse Temporal Pooling Network

Abstract

We propose a weakly supervised temporal action localization algorithm on untrimmed videos using convolutional neural networks. Our algorithm learns from video-level class labels and predicts temporal intervals of human actions with no requirement of temporal localization annotations. We design our network to identify a sparse subset of key segments associated with target actions in a video using an attention module and fuse the key segments through adaptive temporal pooling. Our loss function is comprised of two terms that minimize the video-level action classification error and enforce the sparsity of the segment selection. At inference time, we extract and score temporal proposals using temporal class activations and class-agnostic attentions to estimate the time intervals that correspond to target actions. The proposed algorithm attains state-of-the-art results on the THUMOS14 dataset and outstanding performance on ActivityNet1.3 even with its weak supervision.

AdapNet: Adaptability Decomposing Encoder–Decoder Network for Weakly Supervised Action Recognition and Localization

“…UntrimmedNets [28] employs attention mechanisms to learn the pattern of precut action segments. STPN [29] utilizes a sparsity constraint to detect the activities, which improves the performance of action localization. TSR-Net [30] integrates self-attention and transfer learning with temporal localization framework to obtain precise temporal intervals in untrimmed videos.…”

3C-Net: Category Count and Center Loss for Weakly-Supervised Action Localization

“…Existing approaches have investigated different weak supervision strategies for action localization. The work of [25,14,28] use action category labels in videos for temporal localization, whereas [13] uses point-level supervision to spatio-temporally localize the actions. [17,2] exploit the order of actions in a video as a weak supervision cue.…”

“…Weakly-supervised temporal action localization has been investigated using different types of weak labels, e.g., action categories [25,28,14], movie scripts [12,1] and sparse spatio-temporal points [13]. Recently, Paul et al [16] proposed an action localization approach, demonstrating stateof-the-art results, using video-level category labels as the weak supervision.…”

“…We propose a framework, called 3C-Net, using a novel formulation to learn discriminative action features with enhanced localization capabilities using video-level supervision. As in [14,16], our formulation contains a classification loss term that ensures the inter-class separability of learned features, for video-level action classification. However, this separability at the global video-level alone is insufficient for accurate action localization, which is generally a local temporal-context classification.…”

Weakly-Supervised Completion Moment Detection using Temporal Attention

“…Li et al [19] apply attention for action recognition and action detection in untrimmed sequences, using features from multiple modalities as the input to the temporal attention LSTM before softmax normalisation. Nguyen et al [25] learn attention for action classification. They normalise the attention scores by a sigmoid function, and then use these to estimate the discriminative class-specific temporal regions for localising actions.…”