Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning

Luo, Dezhao; Liu, Chang; Zhou, Yu; Yang, Dongbao; Ma, Can; Ye, Qixiang; Wang, Weiping

doi:10.48550/arxiv.2001.00294

Cited by 8 publications

(33 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With an extra time dimension, videos provides rich static and dynamic information, and there is thus an abundant supply of various supervision signals. A natural way is to extend patch-based context prediction to spatio-temporal scenarios, such as spatio-temporal puzzles [21], video cloze procedure [30] and frame/clip order prediction [26,54,9]. Besides the extension of image based supervisions, recent works propose to learn representations by predicting future frames [12,13].…”

Section: Related Workmentioning

confidence: 99%

Self-supervised Motion Learning from Static Images

Huang

Zhang

Jiang

et al. 2021

Preprint

View full text Add to dashboard Cite

Motions are reflected in videos as the movement of pixels, and actions are essentially patterns of inconsistent motions between the foreground and the background. To well distinguish the actions, especially those with complicated spatio-temporal interactions, correctly locating the prominent motion areas is of crucial importance. However, most motion information in existing videos are difficult to label and training a model with good motion representations with supervision will thus require a large amount of human labour for annotation. In this paper, we address this problem by self-supervised learning. Specifically, we propose to learn Motion from Static Images (MoSI). The model learns to encode motion information by classifying pseudo motions generated by MoSI. We furthermore introduce a static mask in pseudo motions to create local motion patterns, which forces the model to additionally locate notable motion areas for the correct classification. We demonstrate that MoSI can discover regions with large motion even without finetuning on the downstream datasets. As a result, the learned motion representations boost the performance of tasks requiring understanding of complex scenes and motions, i.e., action recognition. Extensive experiments show the consistent and transferable improvements achieved by MoSI. Codes will be soon released.

show abstract

Section: Related Workmentioning

confidence: 99%

Self-supervised Motion Learning from Static Images

Huang

Zhang

Jiang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Evaluation. We adopt the standard evaluation protocol [50,26] during testing. To predict each video sequence, we take 10 video clips uniformly from each testing video sequence and average these prediction results.…”

Section: Implementation Detailsmentioning

confidence: 99%

VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples

Tian

Song

Yang

et al. 2021

Preprint

View full text Add to dashboard Cite

MoCo [11] is effective for unsupervised image representation learning. In this paper, we propose VideoMoCo for unsupervised video representation learning. Given a video sequence as an input sample, we improve the temporal feature representations of MoCo from two perspectives. First, we introduce a generator to drop out several frames from this sample temporally. The discriminator is then learned to encode similar feature representations regardless of frame removals. By adaptively dropping out different frames during training iterations of adversarial learning, we augment this input sample to train a temporally robust encoder. Second, we use temporal decay to model key attenuation in the memory queue when computing the contrastive loss. As the momentum encoder updates after keys enqueue, the representation ability of these keys degrades when we use the current input sample for contrastive learning. This degradation is reflected via temporal decay to attend the input sample to recent keys in the queue. As a result, we adapt MoCo to learn video representations without empirically designing pretext tasks. By empowering the temporal robustness of the encoder and modeling the temporal decay of the keys, our VideoMoCo improves MoCo temporally based on contrastive learning. Experiments on benchmark datasets including UCF101 and HMDB51 show that VideoMoCo stands as a state-of-the-art video representation learning method.

show abstract

“…Although these work demonstrated the effectiveness of selfsupervised representation learning with unlabeled videos and showed impressive performances when transferring the learned features to video recognition tasks, their approaches are only applicable to a CNN that accepts one or two frames as inputs and cannot be applied to network architectures that are suitable for spatio-temporal representations. Therefore, several recent papers [10], [13], [14], [20] used 3D CNNs as backbone networks to learn spatio-temporal representations, among which [10], [13], [14] extended the 2D frame ordering pretext tasks to 3D video clip ordering, and [20] formulated the pretext task as a dense prediction problem and proposed to predict future frames. Very recently, self-supervised learning leveraging multi-modality sources, e.g., learning from video and audio [36], [37], is becoming increasingly popular.…”

Section: Self-supervised Representation Learningmentioning

confidence: 99%

“…Stemmed from these two works, various network architectures are designed to learn video representations, including P3D [46], I3D [1], R(2+1)D [8], etc. In this work, we consider to use three backbone networks, C3D [45], 3D-ResNet [8] and R(2+1)D [8] to validate the proposed approach, following prior works [10], [14]. Backbone networks pre-trained with the proposed spatio-temporal statistics will be used as weight initialization and fine-tuned on UCF101 [47] and HMDB51 [48] datasets for the action recognition downstream task.…”

Section: Representation Learning For Video Analytic Tasksmentioning

confidence: 99%

“…While great progresses have been made, supervised video representation learning is impeded by two major obstacles: (1) Annotation of video data is labour-intensive and expensive, thus restricting supervised learning to relish a large quantity of free video resources on the Internet. (2) Representations learned from labeled video data lack generality and robustness, e.g., video features learned for action recognition do not well to video retrieval task [9], [10].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics

Wang,

Jiao,

Bao

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper proposes a novel pretext task to address the self-supervised video representation learning problem. Specifically, given an unlabeled video clip, we compute a series of spatio-temporal statistical summaries, such as the spatial location and dominant direction of the largest motion, the spatial location and dominant color of the largest color diversity along the temporal axis, etc. Then a neural network is built and trained to yield the statistical summaries given the video frames as inputs. In order to alleviate the learning difficulty, we employ several spatial partitioning patterns to encode rough spatial locations instead of exact spatial Cartesian coordinates. Our approach is inspired by the observation that human visual system is sensitive to rapidly changing contents in the visual field, and only needs impressions about rough spatial locations to understand the visual contents. To validate the effectiveness of the proposed approach, we conduct extensive experiments with several 3D backbone networks, i.e., C3D, 3D-ResNet and R(2+1)D. The results show that our approach outperforms the existing approaches across the three backbone networks on various downstream video analytic tasks including action recognition, video retrieval, dynamic scene recognition, and action similarity labeling. The source code will be made publicly available at: https://github.com/laura-wang/video repres sts.

show abstract

Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning

Cited by 8 publications

References 26 publications

Self-supervised Motion Learning from Static Images

Self-supervised Motion Learning from Static Images

VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples

Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics

Contact Info

Product

Resources

About