Unsupervised Visuomotor Control through Distributional Planning Networks

Yu, Tong; Shevchuk, Gleb; Sadigh, Dorsa; Finn, Chelsea

doi:10.48550/arxiv.1902.05542

Cited by 8 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The idea is to learn a forward model of the world, which forecasts the outcome of an action. In the case of robot control, a popular approach is to learn the state-action transition models in a latent feature embedding space, which are further used for motion planning [8], [9], [10]. Likewise, visual foresight [11] leverages a deep video prediction model to plan the end-effector motion by sampling actions leading to a state which approximates the goal image.…”

Section: Related Workmentioning

confidence: 99%

Introspective Visuomotor Control: Exploiting Uncertainty in Deep Visuomotor Control for Failure Recovery

Hung¹,

Li²,

Wu³

et al. 2021

Preprint

View full text Add to dashboard Cite

End-to-end visuomotor control is emerging as a compelling solution for robot manipulation tasks. However, imitation learning-based visuomotor control approaches tend to suffer from a common limitation, lacking the ability to recover from an out-of-distribution state caused by compounding errors. In this paper, instead of using tactile feedback or explicitly detecting the failure through vision, we investigate using the uncertainty of a policy neural network. We propose a novel uncertainty-based approach to detect and recover from failure cases. Our hypothesis is that policy uncertainties can implicitly indicate the potential failures in the visuomotor control task and that robot states with minimum uncertainty are more likely to lead to task success. To recover from high uncertainty cases, the robot monitors its uncertainty along a trajectory and explores possible actions in the state-action space to bring itself to a more certain state. Our experiments verify this hypothesis and show a significant improvement on task success rate: 12% in pushing, 15% in pick-and-reach and 22% in pick-and-place.

show abstract

Section: Related Workmentioning

confidence: 99%

Introspective Visuomotor Control: Exploiting Uncertainty in Deep Visuomotor Control for Failure Recovery

Hung¹,

Li²,

Wu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In low-dimensional tasks, one can simply take the reward to be the negative 2 -distance in the state space (Andrychowicz et al, 2017). However, defining distance metrics is more challenging in high-dimensional spaces, such as images (Yu et al, 2019). Prior work on visual goal-conditioned RL (Nair et al, 2018;Pong et al, 2019) train an additional state representation model, such as a VAE encoder e VAE : S → Z VAE .…”

Section: Goal-conditioned Reinforcement Learningmentioning

confidence: 99%

Weakly-Supervised Reinforcement Learning for Controllable Behavior

Lee,

Eysenbach,

Salakhutdinov

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks. However, in many settings, an agent must winnow down the inconceivably large space of all possible tasks to the single task that it is currently being asked to solve. Can we instead constrain the space of tasks to those that are semantically meaningful? In this work, we introduce a framework for using weak supervision to automatically disentangle this semantically meaningful subspace of tasks from the enormous space of nonsensical "chaff" tasks. We show that this learned subspace enables efficient exploration and provides a representation that captures distance between states. On a variety of challenging, vision-based continuous control problems, our approach leads to substantial performance gains, particularly as the complexity of the environment grows.

show abstract

“…High-dimensional non-convex optimization problems that have a lot of structure in the solution space naturally arise in the control setting where the controller seeks to optimize the same objective in the same controller dynamical system from different starting states. This has been investigated in, e.g., planning (Ichter et al, 2018;Ichter & Pavone, 2019;Mukadam et al, 2018;Kurutach et al, 2018;Srinivas et al, 2018;Yu et al, 2019;Lynch et al, 2019), and policy distillation (Wang & Ba, 2019). Chandak et al (2019) shows how to learn an action space for model-free learning and Co-Reyes et al (2018); Antonova et al (2019) embed action sequences with a VAE.…”

Section: Rl and Controlmentioning

confidence: 99%

The Differentiable Cross-Entropy Method

Amos,

Yarats

2019

Preprint

View full text Add to dashboard Cite

We study the Cross-Entropy Method (CEM) for the non-convex optimization of a continuous and parameterized objective function and introduce a differentiable variant (DCEM) that enables us to differentiate the output of CEM with respect to the objective function's parameters. In the machine learning setting this brings CEM inside of the end-to-end learning pipeline where this has otherwise been impossible. We show applications in a synthetic energy-based structured prediction task and in non-convex continuous control. In the control setting we show on the simulated cheetah and walker tasks that we can embed their optimal action sequences with DCEM and then use policy optimization to fine-tune components of the controller as a step towards combining model-based and model-free RL.

show abstract

Unsupervised Visuomotor Control through Distributional Planning Networks

Cited by 8 publications

References 0 publications

Introspective Visuomotor Control: Exploiting Uncertainty in Deep Visuomotor Control for Failure Recovery

Introspective Visuomotor Control: Exploiting Uncertainty in Deep Visuomotor Control for Failure Recovery

Weakly-Supervised Reinforcement Learning for Controllable Behavior

The Differentiable Cross-Entropy Method

Contact Info

Product

Resources

About