A Sensorimotor Perspective on Contrastive Multiview Visual Representation Learning

Laflaquière, Alban

doi:10.1109/tcds.2021.3086267

Cited by 3 publications

(4 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…b) Visual data-augmentations: Most contrastive learning approaches use unrealistic data augmentations like color distortion, crop, rotation, resize, blur [8], Jigsaw [18], selection of color channels [19]. In practice, it has been argued that these augmentations are related to a subset of more realistic augmentations [20]: crop can relate to occlusion, resize to object depth motion, color changes to illumination changes etc. Whether unrealistic augmentations and their realistic counterpart truly result in similar representations remains unclear.…”

Section: Related Workmentioning

confidence: 99%

Toddler-inspired embodied vision for learning object representations

Aubret¹,

Teulière²,

Triesch

2022

2022 IEEE International Conference on Development and Learning (ICDL)

View full text Add to dashboard Cite

Recent time-contrastive learning approaches manage to learn invariant object representations without supervision. This is achieved by mapping successive views of an object onto close-by internal representations. When considering this learning approach as a model of the development of human object recognition, it is important to consider what visual input a toddler would typically observe while interacting with objects. First, human vision is highly foveated, with high resolution only available in the central region of the field of view. Second, objects may be seen against a blurry background due to toddlers' limited depth of field. Third, during object manipulation a toddler mostly observes close objects filling a large part of the field of view due to their rather short arms. Here, we study how these effects impact the quality of visual representations learnt through time-contrastive learning. To this end, we let a visually embodied agent "play" with objects in different locations of a near photo-realistic flat. During each play session the agent views an object in multiple orientations before turning its body to view another object. The resulting sequence of views feeds a time-contrastive learning algorithm. Our results show that visual statistics mimicking those of a toddler improve object recognition accuracy in both familiar and novel environments. We argue that this effect is caused by the reduction of features extracted in the background, a neural network bias for large features in the image and a greater similarity between novel and familiar background regions. The results of our model suggest that several influences on toddler's visual input statistics support their unsupervised learning of object representations.

show abstract

Section: Related Workmentioning

confidence: 99%

Toddler-inspired embodied vision for learning object representations

Aubret¹,

Teulière²,

Triesch

2022

2022 IEEE International Conference on Development and Learning (ICDL)

View full text Add to dashboard Cite

show abstract

“…In contrastive ones, models usually learn to embed multiple views of an image into similar representations [4]. The generation process used to obtain these views can be related to some form of action [18], such as cropping which can be linked to head movement and eye saccades. However, these methods include the actions neither to build nor to learn the representations.…”

Section: Related Workmentioning

confidence: 99%

“…They propose to use a pretext task during learning, usually making close the representations of inputs considered similar, to improve performance of a predefined task or in the context of unsupervised learning. In computer vision, some of the similar inputs generation processes can be interpreted as the resulting from movements [18]. In reinforcement learning, temporal prediction of consequences of action is often used as a pretext task.…”

Section: Introductionmentioning

confidence: 99%

The Impact of Action in Visual Representation Learning

Devillers

Chaffraix

Armetta

et al. 2022

2022 IEEE International Conference on Development and Learning (ICDL)

View full text Add to dashboard Cite

Sensori-motor theories, inspired by work in neuroscience, psychology and cognitive science, claim that actions, through learning and mastering of a predictive model, are a key element in the perception of the environment. On the computational side, in the domains of representation learning and reinforcement learning, models are increasingly using self-supervised pretext tasks, such as predictive or contrastive ones, in order to increase the performance on their main task. These pretext tasks are actionrelated even if the action itself is usually not used in the model. In this paper, we propose to study the influence of considering action in the learning of visual representations in deep neural network models, an aspect which is often underestimated w.r.t. sensorimotor theories. More precisely, we quantify two independent factors: 1-whether or not to use the action during the learning of visual characteristics, and 2-whether or not to integrate the action in the representations of the current images. Other aspects will be kept as simple and comparable as possible, that is why we will not consider any specific action policies and combine simple architectures (VAE and LSTM), while using datasets derived from MNIST. In this context, our results show that explicitly including action in the learning process and in the representations improves the performance of the model, which opens interesting perspectives to improve state-of-the-art models of representation learning.

show abstract

“…b) Visual data-augmentations: Most contrastive learning approaches use unrealistic data augmentations like color distortion, crop, rotation, resize, blur [7], [17], Jigsaw [18], selection of color channels [19]. In practice, it has been argued that these augmentations are related to a subset of more realistic augmentations [20]: crop can relate to occlusion, resize to object depth motion, color changes to illumination changes etc. Whether unrealistic augmentations and their realistic counterpart truly result in similar representations remains unclear.…”

Section: Related Workmentioning

confidence: 99%

Embodied vision for learning object representations

Aubret¹,

Teulière²,

Triesch³

2022

Preprint

View full text Add to dashboard Cite

Recent time-contrastive learning approaches manage to learn invariant object representations without supervision. This is achieved by mapping successive views of an object onto close-by internal representations. When considering this learning approach as a model of the development of human object recognition, it is important to consider what visual input a toddler would typically observe while interacting with objects. First, human vision is highly foveated, with high resolution only available in the central region of the field of view. Second, objects may be seen against a blurry background due to infants' limited depth of field. Third, during object manipulation a toddler mostly observes close objects filling a large part of the field of view due to their rather short arms. Here, we study how these effects impact the quality of visual representations learnt through time-contrastive learning. To this end, we let a visually embodied agent "play" with objects in different locations of a near photo-realistic flat. During each play session the agent views an object in multiple orientations before turning its body to view another object. The resulting sequence of views feeds a time-contrastive learning algorithm. Our results show that visual statistics mimicking those of a toddler improve object recognition accuracy in both familiar and novel environments. We argue that this effect is caused by the reduction of features extracted in the background, a neural network bias for large features in the image and a greater similarity between novel and familiar background regions. We conclude that the embodied nature of visual learning may be crucial for understanding the development of human object perception.

show abstract

A Sensorimotor Perspective on Contrastive Multiview Visual Representation Learning

Cited by 3 publications

References 66 publications

Toddler-inspired embodied vision for learning object representations

Toddler-inspired embodied vision for learning object representations

The Impact of Action in Visual Representation Learning

Embodied vision for learning object representations

Contact Info

Product

Resources

About