Acquiring task models for imitation learning through games with a purpose

Kunze, Lars; Haidu, Andrei; Beetz, Michael

doi:10.1109/iros.2013.6696339

Cited by 10 publications

(9 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to collect the ground-truth instruction sequence, we developed an online simulator (motivated by [30]). Our online simulator consists of a virtual 3D environment where the user controls the robot in first-person perspective.…”

Section: Experiments and Resultsmentioning

confidence: 99%

Tell Me Dave: Context-Sensitive Grounding of Natural Language to Manipulation Instructions

Misra

Sung

Lee

et al. 2014

Robotics: Science and Systems X

View full text Add to dashboard Cite

Abstract-We consider performing a sequence of mobile manipulation tasks with instructions given in natural language (NL). Given a new environment, even a simple task such as of boiling water would be performed quite differently depending on the presence, location and state of the objects. We start by collecting a dataset of task descriptions in free-form natural language and the corresponding grounded task-logs of the tasks performed in an online robot simulator. We then build a library of verbenvironment-instructions that represents the possible instructions for each verb in that environment-these may or may not be valid for a different environment and task context. We present a model that takes into account the variations in natural language, and ambiguities in grounding them to robotic instructions with appropriate environment context and task constraints. Our model also handles incomplete or noisy NL instructions. Our model is based on an energy function that encodes such properties in a form isomorphic to a conditional random field. In evaluation, we show that our model produces sequences that perform the task successfully in a simulator and also significantly outperforms the state-of-the-art. We also verify by executing our output instruction sequences on a PR2 robot.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Tell Me Dave: Context-Sensitive Grounding of Natural Language to Manipulation Instructions

Misra

Sung

Lee

et al. 2014

Robotics: Science and Systems X

View full text Add to dashboard Cite

show abstract

“…Playful tools for motion data acquisition based on simulated virtual environments have been explored in the robotics community [11,16]. They offer the benefits of a fully observable world and safety from real world constraints.…”

Section: Gaming Vs Playingmentioning

confidence: 99%

“…It includes complex and diverse manual activities that are easy to perform for (many) people but are hard to describe in a formal or algorithmic way. In robotics, cooking simulators are being investigated for knowledge acquisition [3,5,11,16]. We chose cooking as a paradigmatic scenario for an activity in the household and demonstrate how human computation games that go beyond the playfulness of simulator tools can help to gather data from ordinary persons who have no experience in robotics.…”

Section: Task Domainmentioning

confidence: 99%

See 1 more Smart Citation

Robots, Pancakes, and Computer Games

Walther-Franks

Smeddinck

Szmidt

et al. 2015

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

Self Cite

View full text Add to dashboard Cite

Figure 1: Many robot manipulation tasks (left) are learnt best by observing humans. Human motion can be demonstrated using virtual environments (center). Data acquisition with motion-based games makes demonstrating a fun activity (right). ABSTRACTAutonomous manipulation robots can be valuable aids as interactive agents in the home, yet it has proven extremely difficult to program their behavior. Imitation learning uses data on human demonstrations to build behavioral models for robots. In order to cover a wide range of action strategies, data from many individuals is needed. Acquiring such large amounts of data can be a challenge. Tools for data capturing in this domain must thus implement a good user experience. We propose to use human computation games in order to gather data on human manual behavior. We demonstrate the idea with a strategy game that is operated via a natural user interface. A comparison between using the game for action execution and demonstrating actions in a virtual environment shows that people interact longer and have a better experience when playing the game.

show abstract

“…One method to achieve this personalization is to allow the user to program the system directly, using learning from demonstration. Learning from demonstration has been used successfully to learn manipulation primitives [5], [14], and even watching human action strategies for dual-arm manipulation tasks has proven to be useful in robot controller design [13]. Alternatively, increased personalization can be achieved by allowing the user to modify the retargetting function used to map the user teleoperation sequences onto the robot [6].…”

Section: Related Workmentioning

confidence: 99%

Know thy user: Designing human-robot interaction paradigms for multi-robot manipulation

Lewis

Sukthankar

2014

2014 IEEE/RSJ International Conference on Intelligent Robots and Systems

View full text Add to dashboard Cite

Abstract-This paper tackles the problem of designing an effective user interface for a multi-robot delivery system, composed of robots with wheeled bases and two 3 DOF arms. There are several proven paradigms for increasing the efficacy of human-robot interaction: 1) multimodal interfaces in which the user controls the robots using voice and gesture; 2) configurable interfaces which allow the user to create new commands by demonstrating them; 3) adaptive interfaces which reduce the operator's workload as necessary through increasing robot autonomy. Here we study the relative benefits of configurable vs. adaptive interfaces for multi-robot manipulation. User expertise was measured along three axes (navigation, manipulation, and coordination), and users who performed above threshold on two out of three dimensions on a calibration task were rated as expert. Our experiments reveal that the relative expertise of the user was the key determinant of the best performing interface paradigm for that user, indicating that good user modeling is essential for designing a human-robot interaction system meant to be used for an extended period of time.

show abstract

Acquiring task models for imitation learning through games with a purpose

Cited by 10 publications

References 15 publications

Tell Me Dave: Context-Sensitive Grounding of Natural Language to Manipulation Instructions

Tell Me Dave: Context-Sensitive Grounding of Natural Language to Manipulation Instructions

Robots, Pancakes, and Computer Games

Know thy user: Designing human-robot interaction paradigms for multi-robot manipulation

Contact Info

Product

Resources

About