Temporally Coherent Embeddings for Self-Supervised Video Representation Learning

Knights, Joshua; Harwood, Ben; Ward, Daniel B.; Vanderkop, Anthony; Mackenzie-Ross, Olivia; Moghadam, Peyman

doi:10.1109/icpr48806.2021.9412071

Cited by 20 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This task is generalized as classifying the playback speed of a given modified clip Yao et al [2020], , Jenni et al [2020], Benaim et al [2020], Knights et al [2021]. This task often starts by taking clips of t frames from each video V ∈ R T ×C×H×W and modifying the frame selection in a way that the playback speed is altered.…”

Section: Playback Speedmentioning

confidence: 99%

“…Temporal augmentation has been used in order to extend image-based approaches to video. It is used to generate pairs from modifying the temporal order or the start and end point of a clip interval Knights et al [2020], Lorre et al [2020].…”

Section: Temporal Augmentationmentioning

confidence: 99%

“…In Knights et al [2020], temporal coherency is used by maximizing a similarity function between two temporally adjacent frames in the same video, while minimizing similarity between frames from other videos. The NCE loss uses multiple negative samples that are selected by their embedding distance outside a hypersphere.…”

Section: Temporal Augmentationmentioning

confidence: 99%

“…Some early works in this area have focused on extending works in the image-domain for videos , Tao et al [2020a], Wang et al [2019], Noroozi and Favaro [2016], Ahsan et al [2018], Huo et al [2021], Kim et al [2018b]. Some recent approaches are focusing on the temporal signal so learned representations also capture the temporal dimension of videos Yao et al [2020], , Jenni et al [2020], Benaim et al [2020], Knights et al [2021], Misra et al [2016], , , . While not trivial, work in this area has become increasingly of interest as seen in Figure 2.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Self-Supervised Learning for Videos: A Survey

Schiappa¹,

Rawat²,

Shah³

2022

Preprint

View full text Add to dashboard Cite

The remarkable success of deep learning in various domains relies on the availability of large-scale annotated datasets. However, the use of human-generated annotations leads to models with biased learning, poor domain generalization, and poor robustness. Obtaining annotations is also expensive and requires great effort, which is especially challenging for videos. As an alternative, self-supervised learning provides a way for representation learning which does not require annotations and has shown promise in both image and video domains. Different from the image domain, learning video representations are more challenging due to the temporal dimension, bringing in motion and other environmental dynamics. This also provides opportunities for exclusive ideas which can advance self-supervised learning in the video and multimodal domain. In this survey, we provide a review of existing approaches on self-supervised learning focusing on the video domain. We summarize these methods into three different categories based on their learning objectives: 1) pre-text tasks, 2) generative modeling, and 3) contrastive learning. These approaches also differ in terms of the modality which are being used; 1) video, 2) video-audio, 3) video-text, and 4) video-audio-text. We further introduce the commonly used datasets, downstream evaluation tasks, insights into the limitations of existing works, and the potential future directions in this area.

show abstract

Section: Playback Speedmentioning

confidence: 99%

Section: Temporal Augmentationmentioning

confidence: 99%

Section: Temporal Augmentationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Self-Supervised Learning for Videos: A Survey

Schiappa¹,

Rawat²,

Shah³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…c) Time-contrastive learning for object representations: closer to our work, we emphasize three works that study the representations learnt through realistic transformations. Using videos, some approaches proposed to take advantage of the temporal information to learn image embeddings [27], [28]. But datasets may not simulate the depth of field/foveation and do not allow the study of the complexity of the background.…”

Section: Related Workmentioning

confidence: 99%

Embodied vision for learning object representations

Aubret¹,

Teulière²,

Triesch³

2022

Preprint

View full text Add to dashboard Cite

Recent time-contrastive learning approaches manage to learn invariant object representations without supervision. This is achieved by mapping successive views of an object onto close-by internal representations. When considering this learning approach as a model of the development of human object recognition, it is important to consider what visual input a toddler would typically observe while interacting with objects. First, human vision is highly foveated, with high resolution only available in the central region of the field of view. Second, objects may be seen against a blurry background due to infants' limited depth of field. Third, during object manipulation a toddler mostly observes close objects filling a large part of the field of view due to their rather short arms. Here, we study how these effects impact the quality of visual representations learnt through time-contrastive learning. To this end, we let a visually embodied agent "play" with objects in different locations of a near photo-realistic flat. During each play session the agent views an object in multiple orientations before turning its body to view another object. The resulting sequence of views feeds a time-contrastive learning algorithm. Our results show that visual statistics mimicking those of a toddler improve object recognition accuracy in both familiar and novel environments. We argue that this effect is caused by the reduction of features extracted in the background, a neural network bias for large features in the image and a greater similarity between novel and familiar background regions. We conclude that the embodied nature of visual learning may be crucial for understanding the development of human object perception.

show abstract