Tell me Dave: Context-sensitive grounding of natural language to manipulation           instructions

Misra, Dipendra; Sung, Jaeyong; Lee, Kevin; Saxena, Ashutosh

doi:10.1177/0278364915602060

Cited by 160 publications

(99 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, our goal is to move away from low-level instructions that correspond directly to actions in the environment to high-level task descriptions expressed using complex language. Unlike previous approaches that learn word groundings from pairs of natural language instructions and demonstrations of corresponding high-level actions [22,9] or approaches that try to fill in the gaps when the learned actions do not match [18], our method learns mappings of natural language instructions to task descriptions represented as OO-MDP reward functions, enabling the robot to be autonomous and creative in its solutions to problems that it may never have encountered in training. Our work is most closely related to Howard et al [11]; both approaches map between language and an abstract, goal-based representations that enables creative solutions to problems.…”

Section: Transferring the Learned Model To A Different Robotmentioning

confidence: 99%

Grounding English Commands to Reward Functions

MacGlashan

Babes-Vroman

desJardins

et al. 2015

Robotics: Science and Systems XI

View full text Add to dashboard Cite

Abstract-As intelligent robots become more prevalent, methods to make interaction with the robots more accessible are increasingly important. Communicating the tasks that a person wants the robot to carry out via natural language, and training the robot to ground the natural language through demonstration, are especially appealing approaches for interaction, since they do not require a technical background. However, existing approaches map natural language commands to robot command languages that directly express the sequence of actions the robot should execute. This sequence is often specific to a particular situation and does not generalize to new situations. To address this problem, we present a system that grounds natural language commands into reward functions using demonstrations of different natural language commands being carried out in the environment. Because language is grounded to reward functions, rather than explicit actions that the robot can perform, commands can be high-level, carried out in novel environments autonomously, and even transferred to other robots with different action spaces. We demonstrate that our learned model can be both generalized to novel environments and transferred to a robot with a different action space than the action space used during training.

show abstract

Section: Transferring the Learned Model To A Different Robotmentioning

confidence: 99%

Grounding English Commands to Reward Functions

MacGlashan

Babes-Vroman

desJardins

et al. 2015

Robotics: Science and Systems XI

View full text Add to dashboard Cite

show abstract

“…Similarly, Misra et al (2014) developed a system for mobile robots which is able to learn to ground the language instructions from a corpus of pairs of natural language including both verbs and spatial information. In Yürüten et al (2013), it was proposed that in order to understand the object affordance which can be described by adjectives, the most crucial property is the shape-related one.…”

Section: Language Understanding For Robot Systemsmentioning

confidence: 99%

Sensorimotor input as a language generalisation tool: a neurorobotics model for generation and generalisation of noun-verb combinations with sensorimotor inputs

et al. 2018

View full text Add to dashboard Cite

The paper presents a neurorobotics cognitive model explaining the understanding and generalisation of nouns and verbs combinations when a vocal command consisting of a verb-noun sentence is provided to a humanoid robot. The dataset used for training was obtained from object manipulation tasks with a humanoid robot platform; it includes 9 motor actions and 9 objects placing placed in 6 different locations), which enables the robot to learn to handle real-world objects and actions. Based on the multiple time-scale recurrent neural networks, this study demonstrates its generalisation capability using a large data-set, with which the robot was able to generalise semantic representation of novel combinations of noun-verb sentences, and therefore produce the corresponding motor behaviours. This generalisation process is done via the grounding process: different objects are being interacted, and associated, with different motor behaviours, following a learning approach inspired by developmental language acquisition in infants. Further analyses of the learned network dynamics and representations also demonstrate how the generalisation is possible via the exploitation of this functional hierarchical recurrent network.

show abstract

“…Some pair language and robot actions, then build models for mapping instructions to actions [1,2]. In another study, robots are enabled to learn actions and the lexicons referring to those actions [3].…”

Section: Related Workmentioning

confidence: 99%

A Natural Language Control System for Application Specific Robots

Jajoo¹,

Supekar²,

Hegde³

2017

IJCA

View full text Add to dashboard Cite

Due to the increasing presence of robots in several industries, human-robot interaction using natural language has become an important research area. The concept of controlling robots by transforming language instructions in English into executable code for robots is discussed. The proposed approach makes use of semantic similarity by comparing the given instruction with those within a corpus and executes the instruction most similar to the one given. The method is application specific but the modular nature of the system allows it to be adapted for any robot and for any purpose. For this project, a Raspberry Pi based robot following navigation commands is used for experiments and a success rate of 96% is observed.

show abstract

Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions

Cited by 160 publications

References 55 publications

Grounding English Commands to Reward Functions

Grounding English Commands to Reward Functions

Sensorimotor input as a language generalisation tool: a neurorobotics model for generation and generalisation of noun-verb combinations with sensorimotor inputs

A Natural Language Control System for Application Specific Robots

Contact Info

Product

Resources

About