RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning

Abstract

We study unsupervised video representation learning that seeks to learn both motion and appearance features from unlabeled video only, which can be reused for downstream tasks such as action recognition. This task, however, is extremely challenging due to: 1) the highly complex spatial-temporal information in videos; and 2) the lack of labeled data for training. Unlike the representation learning for static images, it is difficult to construct a suitable self-supervised task to well model both motion and appearance features. More recently, several attempts have been made to learn video representation through video playback speed prediction. However, it is non-trivial to obtain precise speed labels for the videos. More critically, the learnt models may tend to focus on motion pattern and thus may not learn appearance features well. In this paper, we observe that the relative playback speed is more consistent with motion pattern, and thus provide more effective and stable supervision for representation learning. Therefore, we propose a new way to perceive the playback speed and exploit the relative speed between two video clips as labels. In this way, we are able to well perceive speed and learn better motion features. Moreover, to ensure the learning of appearance features, we further propose an appearancefocused task, where we enforce the model to perceive the appearance difference between two video clips. We show that optimizing the two tasks jointly consistently improves the performance on two downstream tasks, namely action recognition and video retrieval. Remarkably, for action recognition on UCF101 dataset, we achieve 93.7% accuracy without the use of labeled data for pre-training, which outperforms the ImageNet supervised pre-trained model.

ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency

“…Specifically, when pretrained with the 3D ResNet-18 backbone, our method outperforms 3D-RotNet [19], ST-Puzzle [21], and DPC [13] by a large margin (80.5% vs. 62.9%, 65.8%, and 68.2%, respectively, on UCF-101 and 52.3% vs. 33.7%, 33.7%, and 34.5%). When utilizing S3D-G as the backbone, our ASC-Net achieves better accuracy than SpeedNet [2], Pace [34], and RSPNet [5] (90.8% vs. 81.1%, 87.1%, and 89.9%, respectively, on UCF-101 and 60.5% vs. 48.8%, 52.6%, and 59.9%) under the same settings. Remarkably, without the need of any annotation for pretraining, our ASCNet outperforms the ImageNet [10] supervised pretrained model over two datasets (90.8% vs. 86.6%, 60.5% vs. 57.7%).…”

“…SpeedNet [2] predicts whether the video clip is sped up or not, while Pace [34] predicts the exact speed of the video clip. Instead of predicting the absolute playback speed, RSPNet [5] predicts the relative speed to avoid the we use a video encoder f (•; θ) to map the clips into appearance and speed embedding space. For the ACP task, we pull the appearance features from the same video closer.…”

“…Self-supervised pretraining stage. Following prior work [34,2,5], we sample 16 consecutive frames with 112 ×112 spatial size for each clip unless specified otherwise. Video clips are augmented using random cropping with resizing, random color jittering, random Gaussian blurring, and random grayscale and solarization.…”

“…Different evaluation protocols. We survey existing self-supervised video representation learning methods and make the following observations about the evaluation protocols: (1) Different works may use different cropping strategies for evaluation, such as center-crop [2,34,5], threecrop [27], and ten-crop [13,14,15] 3, we present the results of our method with different evaluation protocols used in prior works. Impact of the pretraining epochs.…”

ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning

“…Based on the formulation in [31], [15,16] contrast between the representation of the predicted future frames and that of the real ones. Some recent works exploit video pace variation as augmentation and contrast between representations with different paces [39,6]. Whether it is in the video paradigm, which is the focus of this paper, or in the image domain, augmentations are all shown to be critical to learning a strong representation.…”