Video Representation Learning by Recognizing Temporal Transformations

Abstract

We introduce a novel self-supervised learning approach to learn representations of videos that are responsive to changes in the motion dynamics. Our representations can be learned from data without human annotation and provide a substantial boost to the training of neural networks on small labeled data sets for tasks such as action recognition, which require to accurately distinguish the motion of objects. We promote an accurate learning of motion without human annotation by training a neural network to discriminate a video sequence from its temporally transformed versions. To learn to distinguish non-trivial motions, the design of the transformations is based on two principles: 1) To define clusters of motions based on time warps of different magnitude; 2) To ensure that the discrimination is feasible only by observing and analyzing as many image frames as possible. Thus, we introduce the following transformations: forward-backward playback, random frame skipping, and uniform frame skipping. Our experiments show that networks trained with the proposed method yield representations with improved transfer performance for action recognition on UCF101 and HMDB51.

A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning

“…Unsupervised learning in videos has followed a similar trajectory with earlier methods focusing on predictive tasks based on motion, color and spatiotemporal ordering [29,43,1,44,78,85,60,84,58,57,21,51,86,66,22,48,91,16,87,70,45], and contrastive objectives with visual [74,79,34,53,28,92] and audio-visual input [65,4,5,49,3,68,69].…”

Spatio-Temporal Video Representation Learning for AI Based Video Playback Style Prediction

“…It is desired that in the embedding space the distance between two clips from the same video is lesser than the clips from different videos. Jenni et al [18] introduced a novel self-supervised framework to learn video representations which are sensitive to the changes in the motion dynamics. They have observed that the motion of objects is essential for action recognition tasks.…”

Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization

“…In self-supervised video representation learning, a line of works designed various pretext tasks, e.g., temporal ordering [46,74,75], spatiotemporal puzzles [33,63], colorization [59], playback speed prediction [31,6] and temporal cycle-consistency [66,30,37]. Some works proposed to predict future frames from the given sequence to learn feature embeddings [58,57,43,5].…”

“…* Corresponding author. Email: wylin@sjtu.edu.cn To achieve this goal, early works designed various pretext tasks to uncover effective supervision from video sequences [6,46,33,31,74,63]. Recently, contrastive learning has shown to be powerful in image representation learning [28,47,55,12,26,77].…”