Grounded Language Learning: Where Robotics and NLP Meet

Abstract

Grounded language acquisition is concerned with learning the meaning of language as it applies to the physical world. As robots become more capable and ubiquitous, there is an increasing need for non-specialists to interact with and control them, and natural language is an intuitive, flexible, and customizable mechanism for such communication. At the same time, physically embodied agents offer a way to learn to understand natural language in the context of the world to which it refers. This paper gives an overview of the research area, selected recent advances, and some future directions and challenges that remain.

Interactive task learning via embodied corrective feedback

“…In prior ITL tasks, utterances are usually asserted in a context where their intended contents are satisfied (e.g., [38]). Most approaches assume the teacher's utterance is either a request to perform a specific action (e.g., [3]), or it describes the current state (e.g., [42,56,57,60]), or both are supported [48]. This means the non-linguistic context provides a positive example for learning to interpret the teacher's assertion: for instance, the agent can infer from the instruction "Put a red block on a blue block" that there is a red block and a blue block in the current visual scene that satisfy the preconditions of the put action, and its task in updating its model of symbol grounding is to estimate those positive exemplars from that scene, and update its grounding parameters with that positive evidence.…”

“…[49,56,57]). The second approach is to build explicit classifiers for individual words and concepts, which are then combined with methods for joining these classifiers together to provide meanings of extended linguistic expressions via principles of semantic compositionality (e.g., [22,40,42,48,65]). This second approach has been the traditional choice in developing robot systems that learn to interpret instructions [17,22,40] (although see also Karamcheti et al [36], Al-Omari et al [2], Anderson et al [3]).…”

“…. In fact, this could be a strength of using corrective dialogue moves, compared with descriptions that generally do not generate such exemplars except when the linguistic form of the utterance features negation[48].…”

Recognition of cooking activities through air quality sensor data for supporting food journaling

“…In the literature, several novel works have been proposed that employ cutting-edge approaches to address that challenge. For example, authors in [24] centered on the use case of people teaching a robot about objects and tasks in its environment via unconstrained natural language. They designed statistical machine learning approaches to allow robots to gain knowledge about the world from interactions with users, while simultaneously acquiring semantic representations of language about objects and tasks.…”

Transferable Representation Learning in Vision-and-Language Navigation

“…To succeed, VLN agents must internalize the (possibly noisy) natural language instruction, plan action sequences, and move in environments that dynamically change what is presented in their visual fields. These challenging settings bring simulation-based VLN work closer to real-world, language-based interaction with robots [28].…”