Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning

Abstract

We propose a novel self-supervised method, referred to as Video Cloze Procedure (VCP), to learn rich spatial-temporal representations. VCP first generates “blanks” by withholding video clips and then creates “options” by applying spatio-temporal operations on the withheld clips. Finally, it fills the blanks with “options” and learns representations by predicting the categories of operations applied on the clips. VCP can act as either a proxy task or a target task in self-supervised learning. As a proxy task, it converts rich self-supervised representations into video clip operations (options), which enhances the flexibility and reduces the complexity of representation learning. As a target task, it can assess learned representation models in a uniform and interpretable manner. With VCP, we train spatial-temporal representation models (3D-CNNs) and apply such models on action recognition and video retrieval tasks. Experiments on commonly used benchmarks show that the trained models outperform the state-of-the-art self-supervised models with significant margins.

Self-Supervised Visual Learning by Variable Playback Speeds Prediction of a Video

“…HMDB-51 is a dataset that expresses 51 human actions at least 101 times per class and consists of the total 6,849 clips. In this paper, we verify our method by using the clips for pre-training to compare the performance of the selfsupervised learning methods [8], [9], [11]. Specifically, we train a 3D ConvNet using UCF-101 without label information first and fine-tune the model using labeled videos such as UCF-101 and HMDB-51, respectively.…”

“…Xu et al [9] have efficiently improved the frame order prediction method [8] by sorting the order of the neighboring clips as known as video clip order prediction (VCOP), where the clips are consistent with the video dynamics. In [11], the video cloze procedure (VCP) was proposed to learn the spatial-temporal representation of video data based on a method that uses spatial rotations and temporal shuffling method, which enhanced the accuracy in action recognition. Our proposed method is inspired by [8] and [9], but we make use of the playback speeds of the videos, not the correct sequential order of sampled frames [8] or clips [9].…”

“…We conduct experiments on UCF-101 and HMDB-51 and results are measured over three split of each dataset. In Table 8, we compare the proposed method with the latest self-supervised learning methods such as VCOP [9] and VCP [11] for both UCF-101 and HMDB-51. To validate the generality of the proposed method, we evaluate three backbone networks: C3D, R3D, and R(2+1)D in these experiments.…”

“…From the viewpoint of video classification, video data has its inherent characteristics such as spatial and temporal coherence, and most of the video-based self-supervised learning methods leverage spatio-temporal representation learning based on the steady chronological order of video data to understand the underlying temporal dynamics [8], [9], [11]. Specifically, the frames selected at equal temporal intervals are shuffled and their chronological order predicted [8], whereas clips instead of video frames are used to leverage This work is licensed under a Creative Commons Attribution 4.0 License.…”

FC2RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection

“…Due to the uniqueness and discrimination of corners, we look forward to extending this work to accurate arbitrary-shape text detection and end-to-end text spotting. We would also like to combine this work with self-supervised learning [27][28][29][30].…”