Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity

Mandlekar, Ajay; Booher, Jonathan; Spero, Max; Tung, Albert; Gupta, Anchit; Zhu, Yuke; Garg, Animesh; Savarese, Silvio; Li, Feifei

doi:10.48550/arxiv.1911.04052

Cited by 7 publications

(9 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We provide an overview of the most related datasets in Figure 2. Most existing robot datasets, such as MIME [16], DAML [10], RoboTurk [17], [18], and many others [19], [20], [21], [22], [23], [24], [5] only feature a single domain, making them difficult to use for boosting the generalization in other domains. Merging multiple existing datasets into one multi-domain dataset is difficult due to inconsistencies in data collection protocols, time discretization, robot morphologies, and sensors.…”

Section: Related Workmentioning

confidence: 99%

Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets

Ebert¹,

Yang²,

Schmeckpeper³

et al. 2021

Preprint

View full text Add to dashboard Cite

Robot learning holds the promise of learning policies that generalize broadly. However, such generalization requires sufficiently diverse datasets of the task of interest, which can be prohibitively expensive to collect. In other fields, such as computer vision, it is common to utilize shared, reusable datasets, such as ImageNet, to overcome this challenge, but this has proven difficult in robotics. In this paper, we ask: what would it take to enable practical data reuse in robotics for end-to-end skill learning? We hypothesize that the key is to use datasets with multiple tasks and multiple domains, such that a new user that wants to train their robot to perform a new task in a new domain can include this dataset in their training process and benefit from cross-task and cross-domain generalization. To evaluate this hypothesis, we collect a large multi-domain and multi-task dataset, with 7,200 demonstrations constituting 71 tasks across 10 environments, and empirically study how this data can improve the learning of new tasks in new environments. We find that jointly training with the proposed dataset and 50 demonstrations of a never-before-seen task in a new domain on average leads to a 2x improvement in success rate compared to using target domain data alone. We also find that data for only a few tasks in a new domain can bridge the domain gap and make it possible for a robot to perform a variety of prior tasks that were only seen in other domains. These results suggest that reusing diverse multi-task and multi-domain datasets, including our open-source dataset, may pave the way for broader robot generalization, eliminating the need to re-collect data for each new robot learning project.

show abstract

Section: Related Workmentioning

confidence: 99%

Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets

Ebert¹,

Yang²,

Schmeckpeper³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…2: System Overview. We extend the RoboTurk system [26,28] to enable remote users to monitor policy execution and intervene when they would like to take over control using a web browser and a smartphone. The user monitors policy execution by watching a video stream of the robot in their web browser.…”

Section: Operator Intervenes By Moving Phonementioning

confidence: 99%

Human-in-the-Loop Imitation Learning using Remote Teleoperation

Mandlekar¹,

Xu²,

Roberto³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Imitation Learning is a promising paradigm for learning complex robot manipulation skills by reproducing behavior from human demonstrations. However, manipulation tasks often contain bottleneck regions that require a sequence of precise actions to make meaningful progress, such as a robot inserting a pod into a coffee machine to make coffee. Trained policies can fail in these regions because small deviations in actions can lead the policy into states not covered by the demonstrations. Intervention-based policy learning is an alternative that can address this issue -it allows human operators to monitor trained policies and take over control when they encounter failures. In this paper, we build a data collection system tailored to 6-DoF manipulation settings, that enables remote human operators to monitor and intervene on trained policies. We develop a simple and effective algorithm to train the policy iteratively on new data collected by the system that encourages the policy to learn how to traverse bottlenecks through the interventions. We demonstrate that agents trained on data collected by our intervention-based system and algorithm outperform agents trained on an equivalent number of samples collected by non-interventional demonstrators, and further show that our method outperforms multiple state-ofthe-art baselines for learning from the human interventions on a challenging robot threading task and a coffee making task. Additional results and videos at https://sites.google. com/stanford.edu/iwr

show abstract

“…RoboTurk [22,25] is a platform that allows remote users to collect real or simulated task demonstrations through lowlatency teleoperation. Users log in to a website with a realtime video stream of the robot workspace from their robot's unique vantage point, and control their robot's end effector using their smartphone as a 6-DoF motion controller.…”

Section: A Roboturk Overviewmentioning

confidence: 99%

“…Our system can be used "as is" for the collection with real-world robots, similar to Mandlekar et al[22]. However, due to the current COVID measures, we could not get access to our robots.…”

mentioning

confidence: 99%

Learning Multi-Arm Manipulation Through Collaborative Teleoperation

Tung¹,

Wong²,

Mandlekar³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from human demonstrations collected via teleoperation, but has mostly been limited to single-arm manipulation. However, many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. Unfortunately, applying IL to multi-arm manipulation tasks has been challenging -asking a human to control more than one robotic arm can impose significant cognitive burden and is often only possible for a maximum of two robot arms. To address these challenges, we present MULTI-ARM ROBOTURK (MART), a multi-user data collection platform that allows multiple remote users to simultaneously teleoperate a set of robotic arms and collect demonstrations for multi-arm tasks. Using MART, we collected demonstrations for five novel two and three-arm tasks from several geographically separated users. From our data we arrived at a critical insight: most multi-arm tasks do not require global coordination throughout its full duration, but only during specific moments. We show that learning from such data consequently presents challenges for centralized agents that directly attempt to model all robot actions simultaneously, and perform a comprehensive study of different policy architectures with varying levels of centralization on our tasks. Finally, we propose and evaluate a base-residual policy framework that allows trained policies to better adapt to the mixed coordination setting common in multi-arm manipulation, and show that a centralized policy augmented with a decentralized residual model outperforms all other models on our set of benchmark tasks. Additional results and videos at https://roboturk. stanford.edu/multiarm

show abstract

Scaling Robot Supervision to Hundreds of Hours with RoboTurk: Robotic Manipulation Dataset through Human Reasoning and Dexterity

Cited by 7 publications

References 33 publications

Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets

Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets

Human-in-the-Loop Imitation Learning using Remote Teleoperation

Learning Multi-Arm Manipulation Through Collaborative Teleoperation

Contact Info

Product

Resources

About