Generalized Rank Pooling for Activity Recognition

Cherian, Anoop; Fernando, Basura; Harandi, Mehrtash; Gould, Stephen J.

doi:10.1109/cvpr.2017.172

Cited by 80 publications

(92 citation statements)

References 48 publications

(106 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the context of video data, the temporal structure of video data has been exploited to fine-tune networks on train-1 https://github.com/annusha/unsup_temp_embed ing data without labels [34,2]. The temporal ordering of video frames has also been used to learn feature representations for action recognition [20,23,9,4]. Lee et al [20] learn a video representation in an unsupervised manner by solving a sequence sorting problem.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Kukleva

Kuehne

Sener

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

160

View full text Add to dashboard Cite

The task of temporally detecting and segmenting actions in untrimmed videos has seen an increased attention recently. One problem in this context arises from the need to define and label action boundaries to create annotations for training which is very time and cost intensive. To address this issue, we propose an unsupervised approach for learning action classes from untrimmed video sequences. To this end, we use a continuous temporal embedding of framewise features to benefit from the sequential nature of activities. Based on the latent space created by the embedding, we identify clusters of temporal segments across all videos that correspond to semantic meaningful action classes. The approach is evaluated on three challenging datasets, namely the Breakfast dataset, YouTube Instructions, and the 50Salads dataset. While previous works assumed that the videos contain the same high level activity, we furthermore show that the proposed approach can also be applied to a more general setting where the content of the videos is unknown.

show abstract

Section: Related Workmentioning

confidence: 99%

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Kukleva

Kuehne

Sener

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

160

View full text Add to dashboard Cite

show abstract

“…Enforcing such subspace constraints (orthonormality) on these hyperplanes are often empirically seen to demonstrate better performance as is also observed in [10]. The operator ⊙ is the element-wise multiplication and the quantity max(y(θ) ⊙ W ⊤ θ) captures the maximum value of the element-wise multiplication, signifying that if at least one hyperplane classifies θ correctly, then the hinge-loss will be zero.…”

Section: Discriminative Subspace Poolingmentioning

confidence: 88%

“…While, better CNN architectures, such as the recent I3D framework [8], is essential for pushing the state-of-the-art on video tasks, it is also important to have efficient representation learning schemes that can capture the long-term temporal video dynamics from predictions generated by a temporally local model. Recent efforts in this direction, such as rank pooling, temporal segment networks and temporal relation networks [55, 10,22,18,21,5,45], aim to incorporate temporal dynamics over clip-level features. However, such models often ignore the noise in the videos, and use representations that adhere to a plausible criteria.…”

Section: Introductionmentioning

confidence: 99%

Learning Discriminative Video Representations Using Adversarial Perturbations

Wang

Cherian²

2018

Computer Vision – ECCV 2018

Self Cite

View full text Add to dashboard Cite

Adversarial perturbations are noise-like patterns that can subtly change the data, while failing an otherwise accurate classifier. In this paper, we propose to use such perturbations for improving the robustness of video representations. To this end, given a well-trained deep model for per-frame video recognition, we first generate adversarial noise adapted to this model. Using the original data features from the full video sequence and their perturbed counterparts, as two separate bags, we develop a binary classification problem that learns a set of discriminative hyperplanes -as a subspace -that will separate the two bags from each other. This subspace is then used as a descriptor for the video, dubbed discriminative subspace pooling. As the perturbed features belong to data classes that are likely to be confused with the original features, the discriminative subspace will characterize parts of the feature space that are more representative of the original data, and thus may provide robust video representations. To learn such descriptors, we formulate a subspace learning objective on the Stiefel manifold and resort to Riemannian optimization methods for solving it efficiently. We provide experiments on several video datasets and demonstrate state-of-the-art results.

show abstract

“…In Bilen et al, [14], rank pooling is extended towards an early frame-level fusion, dubbed dynamic images; Wang et al [56], extends this idea to use optical flow, which they call dynamic flow representation. Cherian et al [17] generalized rank pooling to include multiple hyperplanes as a subspace, enabling a richer characterization of the spatio-temporal details of the video. This idea was further extended to non-linear feature representations via kernelized rank pooling in [16].…”

Section: Video Representation Using Pooling Schemesmentioning

confidence: 99%

Discriminative Video Representation Learning Using Support Vector Classifiers

Wang

Cherian²

2021

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

Most popular deep models for action recognition in videos generate independent predictions for short clips, which are then pooled heuristically to assign an action label to the full video segment. As not all frames may characterize the underlying action-indeed, many are common across multiple actions-pooling schemes that impose equal importance on all frames might be unfavorable. In an attempt to tackle this problem, we propose discriminative pooling, based on the notion that among the deep features generated on all short clips, there is at least one that characterizes the action. To identify these useful features, we resort to a negative bag consisting of features that are known to be irrelevant, for example, they are sampled either from datasets that are unrelated to our actions of interest or are CNN features produced via random noise as input. With the features from the video as a positive bag and the irrelevant features as the negative bag, we cast an objective to learn a (nonlinear) hyperplane that separates the unknown useful features from the rest in a multiple instance learning formulation within a support vector machine setup. We use the parameters of this separating hyperplane as a descriptor for the full video segment. Since these parameters are directly related to the support vectors in a max-margin framework, they can be treated as a weighted average pooling of the features from the bags, with zero weights given to non-support vectors. Our pooling scheme is end-to-end trainable within a deep learning framework. We report results from experiments on eight computer vision benchmark datasets spanning a variety of video-related tasks and demonstrate state-of-the-art performance across these tasks.Index Terms-video representation, video data mining, discriminative pooling, action recognition, deep learning. ✦• Jue Wang is with the

show abstract

Generalized Rank Pooling for Activity Recognition

Cited by 80 publications

References 48 publications

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Learning Discriminative Video Representations Using Adversarial Perturbations

Discriminative Video Representation Learning Using Support Vector Classifiers

Contact Info

Product

Resources

About