A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions

Abstract

Robots operating alongside humans in diverse, stochastic environments must be able to accurately interpret natural language commands. These instructions often fall into one of two categories: those that specify a goal condition or target state, and those that specify explicit actions, or how to perform a given task. Recent approaches have used reward functions as a semantic representation of goal-based commands, which allows for the use of a state-of-the-art planner to find a policy for the given task. However, these reward functions cannot be directly used to represent action-oriented commands. We introduce a new hybrid approach, the Deep Recurrent Action-Goal Grounding Network (DRAGGN), for task grounding and execution that handles natural language from either category as input, and generalizes to unseen environments. Our robotsimulation results demonstrate that a system successfully interpreting both goaloriented and action-oriented task specifications brings us closer to robust natural language understanding for human-robot interaction.

ELLA: Exploration through Learned Language Abstraction

“…Recent works have made the connection to language: for example, high-level policies can act over the space of low-level instructions (Jiang et al, 2019). Other hierarchical language techniques include learning models that predict symbolic representations of reward functions at different granularities (Arumugam et al, 2017;Karamcheti et al, 2017), and compositional approaches like policy sketches (Andreas et al, 2017) and modular control (Das et al, 2018), methods that explicitly decompose high-level instructions into sequences of predefined low-level instructions, or subtasks.…”

Interactive task learning via embodied corrective feedback

“…The second approach is to build explicit classifiers for individual words and concepts, which are then combined with methods for joining these classifiers together to provide meanings of extended linguistic expressions via principles of semantic compositionality (e.g., [22,40,42,48,65]). This second approach has been the traditional choice in developing robot systems that learn to interpret instructions [17,22,40] (although see also Karamcheti et al [36], Al-Omari et al [2], Anderson et al [3]). In this paper, we adopt the second approach.…”

Zero-shot Learning of Classifiers from Natural Language Quantification

“…Also notable is recent work by Andreas et al (2017), who propose using language descriptions as parameters to model structure in learning tasks in multiple settings. More generally, learning from language has also been previously explored in tasks such as playing games (Branavan et al, 2012), robot navigation (Karamcheti et al, 2017), etc.…”