Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularities

Abstract

Abstract-Humans can ground natural language commands to tasks at both abstract and fine-grained levels of specificity. For instance, a human forklift operator can be instructed to perform a high-level action, like "grab a pallet" or a low-level action like "tilt back a little bit." While robots are also capable of grounding language commands to tasks, previous methods implicitly assume that all commands and tasks reside at a single, fixed level of abstraction. Additionally, methods that do not use multiple levels of abstraction encounter inefficient planning and execution times as they solve tasks at a single level of abstraction with large, intractable state-action spaces closely resembling real world complexity. In this work, by grounding commands to all the tasks or subtasks available in a hierarchical planning framework, we arrive at a model capable of interpreting language at multiple levels of specificity ranging from coarse to more granular. We show that the accuracy of the grounding procedure is improved when simultaneously inferring the degree of abstraction in language used to communicate the task. Leveraging hierarchy also improves efficiency: our proposed approach enables a robot to respond to a command within one second on 90% of our tasks, while baselines take over twenty seconds on half the tasks. Finally, we demonstrate that a real, physical robot can ground commands at multiple levels of abstraction allowing it to efficiently plan different subtasks within the same planning hierarchy.

Sequence-to-Sequence Language Grounding of Non-Markovian Task Specifications

“…The question of how to effectively convert between natural language instructions and robot behavior has been widely studied in previous work [50,34,24,14,9,47,8,18,11,1,33,28,36,27,40,7,2,19,37]. So far, there have been three categories of behavior specifications that these works have mapped natural language to: action sequences, goal states, and LTL specifications.…”

“…The OO-MDP propositional functions serve as possible atoms of the LTL expressions, creating an expressive language for defining tasks. In particular, LTL formulae are sufficiently expressive to subsume semantic representations used in previous goal-based language grounding work such as MacGlashan et al [33] and Arumugam et al [2].…”

Proceedings of the First Workshop on Language Grounding for Robotics

“…There has been a broad and diverse set of work examining how best to interpret and execute natural language instructions on a robot platform (Vogel and Jurafsky, 2010;Tellex et al, 2011;Artzi and Zettlemoyer, 2013;Howard et al, 2014;Andreas and Klein, 2015;MacGlashan et al, 2015;Paul et al, 2016;Mei et al, 2016;Arumugam et al, 2017). Vogel and Jurafsky (2010) produce policies using language and expert trajectories based rewards, which allow for planning within a stochastic environment along with re-planning in case of failure.…”

“…When a reward function is predicted by the model, an MDP planner is applied to derive the resultant policy (see system pipeline Figure 2). We focus our evaluation of all models on the the Cleanup World mobile-manipulator domain (MacGlashan et al, 2015;Arumugam et al, 2017). The Cleanup World domain consists of an agent in a 2-D world with uniquely colored rooms and movable objects.…”

“…MacGlashan et al (2015) and Arumugam et al (2017) leverage some intermediate task representation and then automatically find a low-level trajectory to achieve the goal using a planner. Other approaches, in the action-oriented regime, directly infer action sequences (Tellex et al, 2011;Matuszek et al, 2012;Artzi and Zettlemoyer, 2013;Andreas and Klein, 2015) from the syntactic or semantic parse structure of natural language.…”