Evaluation of Word Representations in Grounding Natural Language Instructions Through Computational Human-Robot Interaction

Abstract

In order to interact with people in a natural way, a robot must be able to link words to objects and actions. Although previous studies in the literature have investigated grounding, they did not consider grounding of unknown synonyms. In this paper, we introduce a probabilistic model for grounding unknown synonymous object and action names using cross-situational learning. The proposed Bayesian learning model uses four different word representations to determine synonymous words. Afterwards, they are grounded through geometric characteristics of objects and kinematic features of the robot joints during action execution. The proposed model is evaluated through an interaction experiment between a human tutor and HSR robot. The results show that semantic and syntactic information both enable grounding of unknown synonyms and that the combination of both achieves the best grounding.

Intention-Related Natural Language Grounding via Object Affordance Detection and Intention Semantic Extraction

“…Paul et al (2018) parse hierarchical abstract and concrete factors from natural language commands and adopts an approximate inference procedure to ground targets within working scenarios. Roesler et al (2019) employ cross-situational learning to ground unknown synonymous objects and actions, and the introduced method utilizes different word representations to identify synonymous words and grounds targets according to the geometric characteristics of targets. These methods are proposed to ground natural language commands which embed specific target objects.…”

Action learning and grounding in simulated human–robot interactions

“…In contrast, another study showed that unknown synonyms, i.e. synonymous words of previously encountered words that have not been encountered before, require semantic and syntactic information to be grounded (Roesler et al, 2019). Since all words appear in several situations, the online grounding mechanism employed in this study uses no additional semantic or syntactic information to ground synonyms.…”

“…The idea of CSL has led to the development of a variety of algorithms that realize CSL in different ways, e.g. through the use of probabilistic models (Aly et al, 2017; Roesler et al, 2019), for grounding of words through percepts in artificial agents. In this section, three CSL algorithms are proposed, which employ CSL in a way that, to the best of our knowledge, has not been proposed or used before.…”

Unsupervised Online Grounding for Social Robots

“…Understanding natural language is non-trivial and requires sophisticated language grounding mechanisms that provide meaning to language by linking words and phrases to corresponding concrete representations, which represent sets of invariant perceptual features obtained through an agent's sensors that are sufficient to distinguish percepts belonging to different concepts [17]. Most grounding research has focused on understanding natural language instructions so that robots can identify and manipulate the correct object [18,19] or navigate to the correct destination [20], while, to the best of our knowledge, no attempts have been made to ground more abstract concepts, such as emotion types, emotion intensities and genders, which are essential to understand natural language texts describing social norms, such as empathy.…”

“…The proposed framework is evaluated through a simulated human-agent interaction experiment in which the agent listens to the speech of different people and receives at the same time a natural language description, describing the gender of the observed person as well as the experienced emotion. Furthermore, the proposed framework is compared to a Bayesian grounding framework that has been employed in several previous studies to ground words through a variety of different percepts [18][19][20][21].…”