Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction

Abstract

Abstract-This paper presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The core issue here is the grounding of referring expressions: infer objects and their relationships from input images and language expressions. INGRESS allows for unconstrained object categories and unconstrained language expressions. Further, it asks questions to disambiguate referring expressions interactively. To achieve these, we take the approach of grounding by generation and propose a two-stage neuralnetwork model for grounding. The first stage uses a neural network to generate visual descriptions of objects, compares them with the input language expression, and identifies a set of candidate objects. The second stage uses another neural network to examine all pairwise relations between the candidates and infers the most likely referred object. The same neural networks are used for both grounding and question generation for disambiguation. Experiments show that INGRESS outperformed a state-of-the-art method on the RefCOCO dataset and in robot experiments with humans.

A Multimodal Target-Source Classifier With Attention Branches to Understand Ambiguous Instructions for Fetching Daily Objects

“…Similarly to many studies, we are interested in understanding fetching instructions in everyday environments. Recent studies have addressed multimodal language understanding (MLU) by using visual semantic embedding for visual grounding [1], [10]- [13], visual question answering [14] or caption generation [15]. This approach embeds the visual and linguistic features into a common latent space.…”

Talk2Car: Taking Control of Your Self-Driving Car

“…This approach allows to ground certain parts of an image with linguistic constituents. Shridhar and Hsu (2018) consider the task where a robot arm has to pick up a certain object based on a given command. This is accomplished by creating captions for extracted regions from a RPN and clustering them together with the original command.…”

Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target–Source Classification

“…Like the authors of many studies in the field of robotics, we are interested in fetching tasks in daily-life environments. Recent studies have handled multimodal language understanding using multimodal similarity-based integration [4]- [7]. The approach proposed in [4] uses an LSTM to learn the probability of a referring expression, while a unified framework for referring expression generation and comprehension was proposed in [5], and introduced to robotics in [6].…”

“…Unfortunately, such systems are timeconsuming and cumbersome especially when considering home environments and non-expert users. Alternatively, recent studies have combined visual and linguistic knowledge by taking a multimodal similarity-based integration National Institute of Information and Communications Technology, 3-5 Hikaridai, Seika, Soraku, Kyoto 619-0289, Japan name.surname@nict.go.jp approach, which uses cosine similarity between linguistic and visual information [4]- [7]. In this approach, visual and linguistic inputs are handled by convolutional neural networks (CNNs) and long short-term memory (LSTM).…”

“…• We propose an MTCM that predicts all target and source pairs given an instruction sentence and a scene. Rather than multimodal similarity-based integration [4]- [7], our method is based on multimodal region-wise classification. The method is detailed in Section IV.…”