GTI: Learning to Generalize across Long-Horizon Tasks from Human Demonstrations

Mandlekar, Ajay; Xu, Danfei; Martí­n-Martí­n, Roberto; Savarese, Silvio; Li, Feifei

doi:10.15607/rss.2020.xvi.061

Cited by 55 publications

(44 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During the demonstration phase, it parses the recorded video to extract the 6 DoF motion trajectory of the manipulated object in the receptacle's coordinate frame. Compared to learning directly in the image pixel space [33], this approach disentangles the object of interest from the background and represents the extracted trajectory independent of any specific scene configuration. This enables the representation to generalize to novel environments, where the initial object and receptacle placement might differ from the demonstration.…”

Section: B Model-free 6 Dof Object Motion Trackingmentioning

confidence: 99%

“…The demonstration video and an example successful robot execution are shown in the first page's figure (top). This task represents commonly considered pickand-place tasks in prior efforts on visual imitation learning [22,34,23,46,60,33,50]. This task is the simplest among the ones considered here but still requires accurate orientation reasoning for stable placement due to the batteries' shape, which is usually long and thin.…”

Section: Battery Standing Taskmentioning

confidence: 99%

See 1 more Smart Citation

You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

Wen¹,

Lian²,

Bekris³

et al. 2022

Preprint

View full text Add to dashboard Cite

Promising results have been achieved recently in category-level manipulation that generalizes across object instances. Nevertheless, it often requires expensive real-world data collection and manual specification of semantic keypoints for each object category and task. Additionally, coarse keypoint predictions and ignoring intermediate action sequences hinder adoption in complex manipulation tasks beyond pick-and-place. This work proposes a novel, category-level manipulation framework that leverages an object-centric, category-level representation and model-free 6 DoF motion tracking. The canonical object representation is learned solely in simulation and then used to parse a category-level, task trajectory from a single demonstration video. The demonstration is reprojected to a target trajectory tailored to a novel object via the canonical representation. During execution, the manipulation horizon is decomposed into longrange, collision-free motion and last-inch manipulation. For the latter part, a category-level behavior cloning (CatBC) method leverages motion tracking to perform closed-loop control. CatBC follows the target trajectory, projected from the demonstration and anchored to a dynamically selected category-level coordinate frame. The frame is automatically selected along the manipulation horizon by a local attention mechanism. This

show abstract

Section: B Model-free 6 Dof Object Motion Trackingmentioning

confidence: 99%

Section: Battery Standing Taskmentioning

confidence: 99%

You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

Wen¹,

Lian²,

Bekris³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…simulation vs. reality). Current imitation learning approaches from pixels are mainly tested within a single domain [23,24,17,38], domains with a limited gap [2,15,34], or require manually labelled data [25]. For example, the third-person imitation work by Stadie et al [34] deals with known actions, observations from the same simulated embodiment, but with a different viewpoint.…”

Section: Related Workmentioning

confidence: 99%

Manipulator-Independent Representations for Visual Imitation

Zhou

Aytar

Bousmalis

2021

Robotics: Science and Systems XVII

View full text Add to dashboard Cite

Imitation learning is an effective tool for robotic learning tasks where specifying a reinforcement learning (RL) reward is not feasible or where the exploration problem is particularly difficult. Imitation, typically behavior cloning or inverse RL, derive a policy from a collection of first-person action-state trajectories. This is contrary to how humans and other animals imitate: we observe a behavior, even from other species, understand its perceived effect on the state of the environment, and figure out what actions our body can perform to reach a similar outcome. In this work, we explore the possibility of third-person visual imitation of manipulation trajectories, only from vision and without access to actions, demonstrated by embodiments different to the ones of our imitating agent. Specifically, we investigate what would be an appropriate representation method with which an RL agent can visually track trajectories of complex manipulation behavior -non-planar with multiple-object interactionsdemonstrated by experts with different embodiments. We present a way to train manipulator-independent representations (MIR) that primarily focus on the change in the environment and have all the characteristics that make them suitable for crossembodiment visual imitation with RL: cross-domain alignment, temporal smoothness, and being actionable. We show that with our proposed method our agents are able to imitate, with complex robot control, trajectories from a variety of embodiments and with significant visual and dynamics differences, e.g. simulationto-reality gap.

show abstract

“…In the imitation learning (IL) context, the control policy is trained using human demonstrations [23]- [27]. This approach provides a safe and efficient way to learn complex skills; however, it is difficult to generalize towards the state space outside the collected demonstration.…”

Section: Related Workmentioning

confidence: 99%

“…Generalization: So far, previous studies tackled the problem of generalization. Several frameworks were proposed to achieve generalizable IL policy in terms of executable tasks and domains [26], [27]. Ebert et al [31] collected a dataset to boost cross-task and cross-domain generalization.…”

Section: Related Workmentioning

confidence: 99%

Ex-DoF: Expansion of Action Degree-of-Freedom with Virtual Camera Rotation for Omnidirectional Image

Tahara¹,

Hirose²

2021

Preprint

View full text Add to dashboard Cite

Inter-robot transfer of training data is a little explored topic in learning and vision-based robot control. Thus, we propose a transfer method from a robot with a lower Degreeof-Freedom (DoF) action to one with a higher DoF utilizing an omnidirectional camera. The virtual rotation of the robot camera enables data augmentation in this transfer learning process. In this study, a vision-based control policy for a 6-DoF robot was trained using a dataset collected by a differential wheeled ground robot with only three DoFs. Towards application of robotic manipulations, we also demonstrate a control system of a 6-DoF arm robot using multiple policies with different fields of view to enable object reaching tasks.

show abstract

GTI: Learning to Generalize across Long-Horizon Tasks from Human Demonstrations

Cited by 55 publications

References 13 publications

You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

Manipulator-Independent Representations for Visual Imitation

Ex-DoF: Expansion of Action Degree-of-Freedom with Virtual Camera Rotation for Omnidirectional Image

Contact Info

Product

Resources

About