Learning Actionable Representations from Visual Observations

Dwibedi, Debidatta; Tompson, Jonathan; Lynch, Corey; Sermanet, Pierre

doi:10.1109/iros.2018.8593951

Cited by 54 publications

(31 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sermanet et al (2017) used metric learning loss for getting a temporally coherent viewpoint invariant embedding space with multi-view or single-view images and used nearest neighbors in the embedding space for imitation learning. Dwibedi et al (2018) extended the approach to multiple frames, and use a temporal cycle consistency (TCC) loss by matching frames over time across videos in Dwibedi et al (2019). Finn and Levine (2016) and Ebert et al (2018) presented a deep action-conditioned visual foresight model with model-predictive control for learning from pixels directly.…”

Section: Related Workmentioning

confidence: 99%

Sequential robot imitation learning from observations

Tanwani

Yan

Calinon

et al. 2021

The International Journal of Robotics Research

View full text Add to dashboard Cite

This paper presents a framework to learn the sequential structure in the demonstrations for robot imitation learning. We first present a family of task-parameterized hidden semi-Markov models that extracts invariant segments (also called sub-goals or options) from demonstrated trajectories, and optimally follows the sampled sequence of states from the model with a linear quadratic tracking controller. We then extend the concept to learning invariant segments from visual observations that are sequenced together for robot imitation. We present Motion2Vec that learns a deep embedding space by minimizing a metric learning loss in a Siamese network: images from the same action segment are pulled together while being pushed away from randomly sampled images of other segments, and a time contrastive loss is used to preserve the temporal ordering of the images. The trained embeddings are segmented with a recurrent neural network, and subsequently used for decoding the end-effector pose of the robot. We first show its application to a pick-and-place task with the Baxter robot while avoiding a moving obstacle from four kinesthetic demonstrations only, followed by suturing task imitation from publicly available suturing videos of the JIGSAWS dataset with state-of-the-art [Formula: see text]% segmentation accuracy and [Formula: see text] cm error in position per observation on the test set.

show abstract

Section: Related Workmentioning

confidence: 99%

Sequential robot imitation learning from observations

Tanwani

Yan

Calinon

et al. 2021

The International Journal of Robotics Research

View full text Add to dashboard Cite

show abstract

“…The sample-efficiency gains from reconstruction-based auxiliary losses have been benchmarked in [3,29,37]. Recently, contrastive learning has been used to extract reward signals in the latent space [38][39][40]; and study representation learning on Atari games [41].…”

Section: Auxiliary Tasksmentioning

confidence: 99%

An Experimental Study on State Representation Extraction for Vision-Based Deep Reinforcement Learning

Ren¹,

Zeng²,

Zhou³

et al. 2021

Applied Sciences

View full text Add to dashboard Cite

Scaling end-to-end learning to control robots with vision inputs is a challenging problem in the field of deep reinforcement learning (DRL). While achieving remarkable success in complex sequential tasks, vision-based DRL remains extremely data-inefficient, especially when dealing with high-dimensional pixels inputs. Many recent studies have tried to leverage state representation learning (SRL) to break through such a barrier. Some of them could even help the agent learn from pixels as efficiently as from states. Reproducing existing work, accurately judging the improvements offered by novel methods, and applying these approaches to new tasks are vital for sustaining this progress. However, the demands of these three aspects are seldom straightforward. Without significant criteria and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the previous methods are meaningful. For this reason, we conducted ablation studies on hyperparameters, embedding network architecture, embedded dimension, regularization methods, sample quality and SRL methods to compare and analyze their effects on representation learning and reinforcement learning systematically. Three evaluation metrics are summarized, including five baseline algorithms (including both value-based and policy-based methods) and eight tasks are adopted to avoid the particularity of each experiment setting. We highlight the variability in reported methods and suggest guidelines to make future results in SRL more reproducible and stable based on a wide number of experimental analyses. We aim to spur discussion about how to assure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

show abstract

“…In RL, self-supervision also gained momentum in recent years [21,43,48], with temporal information being featured [1]. Notably, several works [3,12,20,42] leverage temporal consistency to learn useful representations, effectively learning to discriminate between observations that are temporally close and observations that are temporally distant. In comparison to all these works, we estimate the arrow of time through temporal order prediction with the explicit goal of finding irreversible transitions or actions.…”

Section: Related Workmentioning

confidence: 99%

There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning

Grinsztajn¹,

Ferret²,

Pietquin³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose to learn to distinguish reversible from irreversible actions for better informed decision-making in Reinforcement Learning (RL). From theoretical considerations, we show that approximate reversibility can be learned through a simple surrogate task: ranking randomly sampled trajectory events in chronological order. Intuitively, pairs of events that are always observed in the same order are likely to be separated by an irreversible sequence of actions. Conveniently, learning the temporal order of events can be done in a fully self-supervised way, which we use to estimate the reversibility of actions from experience, without any priors. We propose two different strategies that incorporate reversibility in RL agents, one strategy for exploration (RAE) and one strategy for control (RAC). We demonstrate the potential of reversibility-aware agents in several environments, including the challenging Sokoban game. In synthetic tasks, we show that we can learn control policies that never fail and reduce to zero the side-effects of interactions, even without access to the reward function.* Equal contribution.

show abstract

Learning Actionable Representations from Visual Observations

Cited by 54 publications

References 35 publications

Sequential robot imitation learning from observations

Sequential robot imitation learning from observations

An Experimental Study on State Representation Extraction for Vision-Based Deep Reinforcement Learning

There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning

Contact Info

Product

Resources

About