Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces

Valassakis, Eugene; Palo, Norman Di; Johns, Edward J.

doi:10.48550/arxiv.2105.11283

Cited by 1 publication

(1 citation statement)

References 24 publications

(42 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A wide range of works have focused on algorithmic development for end-to-end learning of vision-based object manipulation skills Agrawal et al, 2016;Kalashnikov et al, 2018;Srinivas et al, 2018;Zhu et al, 2018;Jayaraman et al, 2018;Rafailov et al, 2021). Some works on learned visuomotor control use eye-in-hand cameras for tasks such as grasping (Song et al, 2020) and insertion Puang et al, 2020;Luo et al, 2021;Valassakis et al, 2021), and others which pre-date end-to-end visuomotor learning use both eye-in-hand and third-person cameras for visual servoing (Flandin et al, 2000;Lippiello et al, 2005). Very few works consider the design of camera placements (Zaky et al, 2020) or conduct any controlled comparisons on different combinations of visual perspectives (Zhan et al, 2020;Mandlekar et al, 2021;Wu et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Vision-Based Manipulators Need to Also See from Their Hands

Hsu¹,

Jin²,

Rafailov³

et al. 2022

Preprint

View full text Add to dashboard Cite

We study how the choice of visual perspective affects learning and generalization in the context of physical manipulation from raw sensor observations. Compared with the more commonly used global third-person perspective, a hand-centric (eye-in-hand) perspective affords reduced observability, but we find that it consistently improves training efficiency and out-of-distribution generalization. These benefits hold across a variety of learning algorithms, experimental settings, and distribution shifts, and for both simulated and real robot apparatuses. However, this is only the case when hand-centric observability is sufficient; otherwise, including a third-person perspective is necessary for learning, but also harms outof-distribution generalization. To mitigate this, we propose to regularize the thirdperson information stream via a variational information bottleneck. On six representative manipulation tasks with varying hand-centric observability adapted from the Meta-World benchmark, this results in a state-of-the-art reinforcement learning agent operating from both perspectives improving its out-of-distribution generalization on every task. While some practitioners have long put cameras in the hands of robots, our work systematically analyzes the benefits of doing so and provides simple and broadly applicable insights for improving end-to-end learned vision-based robotic manipulation. 1 Figure 1: Illustration suggesting the role that visual perspective can play in facilitating the acquisition of symmetries with respect to certain transformations on the world state s. T0: planar translation of the end-effector and cube. T1: vertical translation of the table surface, end-effector, and cube. T2: addition of distractor objects. O3: third-person perspective. O h : hand-centric perspective.

show abstract