Learning Spatiotemporal 3D Convolution with Video Order Self-supervision

Suzuki, Tomoyuki; Itazuri, Takahiro; Hara, Kensho; Kataoka, Hirokatsu

doi:10.1007/978-3-030-11012-3_45

Cited by 9 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pretext task based approaches: The work by Misra et al [35] was one of the first attempts at self-supervised video representation learning by learning to verify correct frame order. Other pretext tasks based on correct temporal order learning include: identifying the correctly ordered tuple from a set of shuffled orderings [13,41], sorting frame order [29], and predicting clip order [52]. There are some methods which extend pretext tasks from the image domain to video domain, for example, solving spatiotemporal jigsaw puzzles [3,26,4], identifying the rotation of transformed video clips [22].…”

Section: Related Workmentioning

confidence: 99%

TCLR: Temporal Contrastive Learning for Video Representation

Dave,

Gupta,

Rizve

et al. 2021

Preprint

View full text Add to dashboard Cite

Contrastive learning has nearly closed the gap between supervised and self-supervised learning of image representations. Existing extensions of contrastive learning to the domain of video data however do not explicitly attempt to represent the internal distinctiveness across the temporal dimension of video clips. We develop a new temporal contrastive learning framework consisting of two novel losses to improve upon existing contrastive self-supervised video representation learning methods. The first loss adds the task of discriminating between non-overlapping clips from the same video, whereas the second loss aims to discriminate between timesteps of the feature map of an input clip in order to increase the temporal diversity of the features. Temporal contrastive learning achieves significant improvement over the state-of-the-art results in downstream video understanding tasks such as action recognition, limited-label action classification, and nearest-neighbor video retrieval on video datasets across multiple 3D CNN architectures. With the commonly used 3D-ResNet-18 architecture, we achieve 82.4% (+5.1% increase over the previous best) top-1 accuracy on UCF101 and 52.9% (+5.4% increase) on HMDB51 action classification, and 56.2% (+11.7% increase) Top-1 Recall on UCF101 nearest neighbor video retrieval.

show abstract

Section: Related Workmentioning

confidence: 99%

TCLR: Temporal Contrastive Learning for Video Representation

Dave,

Gupta,

Rizve

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…While modeling time remains a challenge, it also presents a natural source of supervision that has been exploited for self-supervised learning. For example, as a proxy signal by posing pretext tasks involving spatio-temporal jigsaw [1,43,52], video speed [10,16,47,94,109,123], arrow of time [78,80,112], frame/clip ordering [24,70,90,97,116], video continuity [60], or tracking [44,106,111]. Several works have also used contrastive learning to obtain spatio-temporal representations by (i) contrasting temporally augmented versions of a clip [46,77,81], or (ii) encouraging consistency between local and global temporal contexts [9,17,85,122].…”

Section: Time In Visionmentioning

confidence: 99%

Test of Time: Instilling Video-Language Models with a Sense of Time

Bagad¹,

Tapaswi²,

Snoek³

2023

Preprint

View full text Add to dashboard Cite

Modeling and understanding time remains a challenge in contemporary video understanding models. With language emerging as a key driver towards powerful generalization, it is imperative for foundational video-language models to have a sense of time. In this paper, we consider a specific aspect of temporal understanding: consistency of time order as elicited by before/after relations. We establish that six existing video-language models struggle to understand even such simple temporal relations. We then question whether it is feasible to equip these foundational models with temporal awareness without re-training them from scratch. Towards this, we propose a temporal adaptation recipe on top of one such model, VideoCLIP, based on post-pretraining on a small amount of video-text data. We conduct a zero-shot evaluation of the adapted models on six datasets for three downstream tasks which require a varying degree of time awareness. We observe encouraging performance gains especially when the task needs higher time awareness. Our work serves as a first step towards probing and instilling a sense of time in existing video-language models without the need for data and compute-intense training from scratch.

show abstract

“…These tasks operate multiple transformations on source videos for model to recognize and have shown to be effective in self-supervised representation learning (Wang et al 2021b). Examples include identifying temporal order of shuffled clips or frames (Lee et al 2017;Xu et al 2019;Suzuki et al 2018), predicting video's playback rate (Benaim et al 2020;Wang, Jiao, and Liu 2020;Chen et al 2021) or motion and appearance statistics (Wang et al 2019), identifying the rotation angle of video clips (Jing et al 2018) or solving spatiotemporal jigsaw puzzles (Ahsan, Madhok, and Essa 2019;Kim, Cho, and Kweon 2019), etc. In this work, we focus on an essential yet less-touched video property, the video continuity.…”

Section: Related Workmentioning

confidence: 99%

Self-Supervised Spatiotemporal Representation Learning by Exploiting Video Continuity

Liang

Quader

Chi

et al. 2022

AAAI

View full text Add to dashboard Cite

Recent self-supervised video representation learning methods have found significant success by exploring essential properties of videos, e.g. speed, temporal order, etc. This work exploits an essential yet under-explored property of videos, the \textit{video continuity}, to obtain supervision signals for self-supervised representation learning. Specifically, we formulate three novel continuity-related pretext tasks, i.e. continuity justification, discontinuity localization, and missing section approximation, that jointly supervise a shared backbone for video representation learning. This self-supervision approach, termed as Continuity Perception Network (CPNet), solves the three tasks altogether and encourages the backbone network to learn local and long-ranged motion and context representations. It outperforms prior arts on multiple downstream tasks, such as action recognition, video retrieval, and action localization. Additionally, the video continuity can be complementary to other coarse-grained video properties for representation learning, and integrating the proposed pretext task to prior arts can yield much performance gains.

show abstract

Learning Spatiotemporal 3D Convolution with Video Order Self-supervision

Cited by 9 publications

References 7 publications

TCLR: Temporal Contrastive Learning for Video Representation

TCLR: Temporal Contrastive Learning for Video Representation

Test of Time: Instilling Video-Language Models with a Sense of Time

Self-Supervised Spatiotemporal Representation Learning by Exploiting Video Continuity

Contact Info

Product

Resources

About