CLIPort: What and Where Pathways for Robotic Manipulation

Abstract

How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPORT, a language-conditioned imitationlearning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies. † Work done partly while the author was a part-time intern at NVIDIA.

Interactive Visual Task Learning for Robots

“…Moreover, our approach can learn a concept hierarchy starting from zero known concepts, displaying the adaptability of our model under a continual learning setup. Previous work has focused on language conditioned manipulation (Shridhar, Manuelli, and Fox 2021;Liu et al 2021;Brohan et al 2023b,a). Shridhar, Manuelli, and Fox 2021 computes a pick and place location conditioned on linguistic and visual inputs.…”

“…Human Subjects Experiment Domain We evaluate our approach with the human-subjects experiment in a 2-D object rearrangement domain, which is a problem commonly used in language grounding and HRI research (Liu et al 2021;Shridhar, Manuelli, and Fox 2021). The domain we choose for this study is the House-Construction domain which we introduce in Domains Section.…”

“…Previously, large image and language models have been extended to robotics to manipulate novel objects, and create visual scenes (Shridhar, Manuelli, and Fox 2021;Brohan et al 2023a). These methods recognize novel objects by using their underlying large language and visual models to extract task-relevant knowledge.…”

Robotic Grasping from Classical to Modern: A Survey

“…They demonstrated that the final goal-conditioned policy performed extremely well. [223] proposed CLIPort, also based on a large-scale unsupervised representation learning model named CLIP [196], which demonstrated surprisingly good generality when trained with a few demonstrations for natural-language-conditioned robotic manipulation.…”

Image Segmentation Using Text and Image Prompts

“…CLIP Extensions Despite CLIP [8] being fairly new, multiple derivative works across different sub-fields have emerged. CLIP was combined with a GAN to modify images based on a text prompt [43] and in robotics to generalize to unseen objects in manipulations tasks [44]. Other work focused on understanding CLIP in more detail.…”