Temporally Coherent Embeddings for Self-Supervised Video Representation Learning

Abstract

This paper presents TCE: Temporally Coherent Embeddings for self-supervised video representation learning. The proposed method exploits inherent structure of unlabeled video data to explicitly enforce temporal coherency in the embedding space, rather than indirectly learning it through ranking or predictive pretext tasks. In the same way that high-level visual information in the world changes smoothly, we believe that nearby frames in learned representations should demonstrate similar properties. Using this assumption, we train the TCE model to encode videos such that adjacent frames exist close to each other and videos are separated from one another. Using TCE we learn robust representations from large quantities of unlabeled video data. We evaluate our self-supervised trained TCE model by adding a classification layer and finetuning the learned representation on the downstream task of video action recognition on the UCF101 dataset. We obtain 67.01% accuracy and outperform the state-of-the-art self-supervised methods despite using a significantly smaller dataset for pre-training. Notably, we demonstrate results competitive with more complex 3D-CNN based networks while training with a 2D-CNN network backbone on action recognition tasks. The code and pre-trained models for this paper can be downloaded at http://github.com/csiro-robotics/tce.git.

TCLR: Temporal Contrastive Learning for Video Representation

“…For example, using video clips with different playback rates as positive pairs for contrastive loss along with predicting the playback rate [49]. Other works propose frame-based contrastive learning, along with existing pretext tasks of frame rotation prediction [27] and frame-tuple order verification [55], with the pretext tasks borrowed from from [22] and [35], respectively. Unlike previous work, we take a different approach and propose TCLR: Temporal Contrastive Learning for Video Representation, to improve video contrastive learning by adding temporal contrastive losses that focus on the harder task of discriminating between clips within the same video instance.…”

Self-supervised learning through the eyes of a child

“…Effectively, this model treats temporal jitter between neighboring frames as another type of data augmentation. A similar temporal contrastive learning model was proposed by Knights et al (2020) before.…”

“…by applying color distortions to an image (Chen et al, 2020b). Contrastive self-supervised learning has been applied to both images (Oord et al, 2018;Hjelm et al, 2018;He et al, 2019;Chen et al, 2020b,a) and videos (Sermanet et al, 2018;Zhuang et al, 2020;Knights et al, 2020) with promising results. However, these works were primarily motivated by computer vision applications, and did not apply self-supervised learning methods to a developmentally realistic, longitudinal, first-person video dataset.…”

Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast

“…[8], [9] employ contrastive loss for representation learning and achieve high accuracy on classification and segmentation tasks. [10], [14] use the temporal correlations in the streaming data to improve representation learning. However, all these works assume that the whole training dataset is available in the learning process, and each mini-batch can be formed by sampling from the dataset.…”