Spatiotemporal Contrastive Video Representation Learning

Qian, Rui; Meng, Tianjian; Gong, Boqing; Yang, Ming–Hsuan; Wang, Huisheng; Belongie, Serge; Cui, Yin

doi:10.48550/arxiv.2008.03800

Cited by 48 publications

(67 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most works demonstrate that selecting robust pretext tasks along with suitable data augmentations can greatly boost the quality of representations. In the image domain, data augmentation involves mainly color transformation and geometric transformations such as cropping, resizing, rotation, and flipping [3][6][45] [46][47] [48]. Recently, SwAV [10] outperformed other self-supervised methods by using multiple augmentations.…”

Section: B Contrastive Learningmentioning

confidence: 99%

Contrastive Embedding Distribution Refinement and Entropy-Aware Attention for 3D Point Cloud Classification

Feng¹,

Cao²,

Xue³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning a powerful representation from point clouds is a fundamental and challenging problem in the field of computer vision. Different from images where RGB pixels are stored in the regular grid, for point clouds, the underlying semantic and structural information of point clouds is the spatial layout of the points. Moreover, the properties of challenging incontext and background noise pose more challenges to point cloud analysis. One assumption is that the poor performance of the classification model can be attributed to the indistinguishable embedding feature that impedes the search for the optimal classifier. This work offers a new strategy for learning powerful representations via a contrastive learning approach that can be embedded into any point cloud classification network. First, we propose a supervised contrastive classification (SCC) method to implement embedding feature distribution refinement by improving the intra-class compactness and inter-class separability. Second, to solve the confusion problem caused by small inter-class variations between some similar-looking categories, we propose a confusion-prone class mining (CPCM) strategy to alleviate the confusion effect. Finally, considering that outliers of the sample clusters in the embedding space may cause performance degradation, we design an entropy-aware attention (EAA) module with information entropy theory to identify the outlier cases and the unstable samples by measuring the uncertainty of predicted probability. We assume that the outlier and unstable samples need to be ignored and enhanced, respectively, to enable the model to learn more robust decision boundaries. The results of extensive experiments demonstrate that our method outperforms the state-of-the-art approaches by achieving 82.9% accuracy on the real-world ScanObjectNN dataset and substantial performance gains up to 2.9% in DCGNN, 3.1% in PointNet++, and 2.4% in GBNet. The code will be available at https://github.com/YangFengSEU/CEDR.

show abstract

Section: B Contrastive Learningmentioning

confidence: 99%

Contrastive Embedding Distribution Refinement and Entropy-Aware Attention for 3D Point Cloud Classification

Feng¹,

Cao²,

Xue³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Several Selfsupervised methods [2,15] for video tasks have been studied by the computer vision community. Qian proposed [26] self-supervised Contrastive Video Representation Learning (CVRL) method which uses the contrastive loss to map the video clips in the embedding space. It is desired that in the embedding space the distance between two clips from the same video is lesser than the clips from different videos.…”

Section: Related Workmentioning

confidence: 99%

Spatio-Temporal Video Representation Learning for AI Based Video Playback Style Prediction

Parihar,

Ramola,

Saha

et al. 2021

Preprint

View full text Add to dashboard Cite

Ever-increasing smartphone-generated video content demands intelligent techniques to edit and enhance videos on power-constrained devices. Most of the best performing algorithms for video understanding tasks like action recognition, localization, etc., rely heavily on rich spatiotemporal representations to make accurate predictions. For effective learning of the spatio-temporal representation, it is crucial to understand the underlying object motion patterns present in the video. In this paper, we propose a novel approach for understanding object motions via motion type classification. The proposed motion type classifier predicts a motion type for the video based on the trajectories of the objects present. Our classifier assigns a motion type for the given video from the following five primitive motion classes: linear, projectile, oscillatory, local and random. We demonstrate that the representations learned from the motion type classification generalizes well for the challenging downstream task of video retrieval. Further, we proposed a recommendation system for video playback style based on the motion type classifier predictions.

show abstract

“…Recent advances on contrastive learning for image representation learning [8,57] also show promising results on videos. Research along this direction has achieved the stateof-the-art by contrasting on the temporal dimension [17,47], distilling motion representations [18], and designing better video augmentations [43].…”

Section: Related Workmentioning

confidence: 99%

Towards Tokenized Human Dynamics Representation

Li¹,

Xiao²,

Wu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Spatiotemporal Contrastive Video Representation Learning

Cited by 48 publications

References 60 publications

Contrastive Embedding Distribution Refinement and Entropy-Aware Attention for 3D Point Cloud Classification

Contrastive Embedding Distribution Refinement and Entropy-Aware Attention for 3D Point Cloud Classification

Spatio-Temporal Video Representation Learning for AI Based Video Playback Style Prediction

Towards Tokenized Human Dynamics Representation

Contact Info

Product

Resources

About