Improving Grounded Natural Language Understanding through Human-Robot Dialog

Abstract

Natural language understanding for robotics can require substantial domain-and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamicallycontinually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world.

Intention-Related Natural Language Grounding via Object Affordance Detection and Intention Semantic Extraction

“…Ahn et al (2018) utilize position maps generated by the hourglass network (Newell et al, 2016) and a question generation module to infer referred objects. Thomason et al (2019) translate spoken language instructions into robot action commands and uses clarification conversations with human users to ground targets. However, conversation and dialog systems make HRI time-consuming and cumbersome.…”

Your instruction may be crisp, but not clear to me!

“…Dialogue agents for task execution by robots are mostly focused on eliciting missing information [7], [8], knowledge grounding [15], [16] and interactive task learning [17], [18]. To the best of our knowledge, dialogue to resolve task prediction failures due to ambiguity and novelty in the instruction is not well investigated.…”

Building Language-Agnostic Grounded Language Learning Systems

“…There have been a number of successful approaches in different areas, such as navigation [31], [20], understanding commands and directions [2], [1] or action words [5], grounding spatial relations and concepts [24], and referring expressions for objects in images [8], [36]. Other work has explored interactively grounding additional non-visual properties like sound and weight [32].…”