Crossmodal Language Grounding in an Embodied Neurocognitive Model

Heinrich, S.; Yao, Yuan; Hinz, Tobias; Liu, Zhiyuan; Hummel, Thomas; Kerzel, Matthias; Weber, Cornelius; Wermter, Stefan

doi:10.3389/fnbot.2020.00052

Cited by 24 publications

(19 citation statements)

References 63 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The recent review by Uc-Cetina et al (2021) illustrates the applicability of RL in NLP to some extent, such as machine translation, language understanding, and text generation. The authors also suggest considering embodiment (Heinrich et al, 2020), textual domain knowledge, and conversational settings. Bisk et al (2020) focus further on embodiment and highlight the importance of physical and social context, more precisely, multimodal sensory experiences, to apprehend the coherency of words and actions.…”

Section: Reinforcement Learning and Computational Language Understanding Methodsmentioning

confidence: 99%

“…However, models like GPT-3 and DALL-E consider only disembodied language learning without any sensorimotor grounding because, unlike robots, they cannot physically interact with the world. Insights for grounded language learning in robotics (Heinrich et al, 2020) with sequential decision-making settings (Akakzia et al, 2021;Lynch and Sermanet, 2021) and embodied cognition (Feldman and Narayanan, 2004;Fischer and Zwaan, 2008) accentuate the need for embodied grounding. This includes physical interaction and multiple sensory modalities to develop systems that understand language more like humans (Anderson, 1972;Wermter et al, 2009;McClelland et al, 2020).…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

Röder¹,

Özdemir²,

Nguyen³

et al. 2021

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Human language is inherently embodied and grounded in sensorimotor representations of the self and the world around it. This suggests that the body schema and ideomotor action-effect associations play an important role in language understanding, language generation, and verbal/physical interaction with others. There are computational models that focus purely on non-verbal interaction between humans and robots, and there are computational models for dialog systems that focus only on verbal interaction. However, there is a lack of research that integrates these approaches. We hypothesize that the development of computational models of the self is very appropriate for considering joint verbal and physical interaction. Therefore, they provide the substantial potential to foster the psychological and cognitive understanding of language grounding, and they have significant potential to improve human-robot interaction methods and applications. This review is a first step toward developing models of the self that integrate verbal and non-verbal communication. To this end, we first analyze the relevant findings and mechanisms for language grounding in the psychological and cognitive literature on ideomotor theory. Second, we identify the existing computational methods that implement physical decision-making and verbal interaction. As a result, we outline how the current computational methods can be used to create advanced computational interaction models that integrate language grounding with body schemas and self-representations.

show abstract

Section: Reinforcement Learning and Computational Language Understanding Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

Röder¹,

Özdemir²,

Nguyen³

et al. 2021

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Most of these models perform unidirectionally. Our approach aims to map language commands to actions and perceived actions to language, as in the embodied neurocognitive model of Heinrich et al ( 2020 ). Other contribution has been made to address language learning (Morse and Cangelosi, 2017 ), motor learning (Demiris and Khadhouri, 2006 ), and affordance learning (Stoytchev, 2008 ; Jamone et al, 2018 ), by successful integration of language and the physical body.…”

Section: Related Workmentioning

confidence: 99%

Learning Actions From Natural Language Instructions Using an ON-World Embodied Cognitive Architecture

2021

View full text Add to dashboard Cite

Endowing robots with the ability to view the world the way humans do, to understand natural language and to learn novel semantic meanings when they are deployed in the physical world, is a compelling problem. Another significant aspect is linking language to action, in particular, utterances involving abstract words, in artificial agents. In this work, we propose a novel methodology, using a brain-inspired architecture, to model an appropriate mapping of language with the percept and internal motor representation in humanoid robots. This research presents the first robotic instantiation of a complex architecture based on the Baddeley's Working Memory (WM) model. Our proposed method grants a scalable knowledge representation of verbal and non-verbal signals in the cognitive architecture, which supports incremental open-ended learning. Human spoken utterances about the workspace and the task are combined with the internal knowledge map of the robot to achieve task accomplishment goals. We train the robot to understand instructions involving higher-order (abstract) linguistic concepts of developmental complexity, which cannot be directly hooked in the physical world and are not pre-defined in the robot's static self-representation. Our proposed interactive learning method grants flexible run-time acquisition of novel linguistic forms and real-world information, without training the cognitive model anew. Hence, the robot can adapt to new workspaces that include novel objects and task outcomes. We assess the potential of the proposed methodology in verification experiments with a humanoid robot. The obtained results suggest robust capabilities of the model to link language bi-directionally with the physical environment and solve a variety of manipulation tasks, starting with limited knowledge and gradually learning from the run-time interaction with the tutor, past the pre-trained stage.

show abstract

“…Heinrich et al (2019) provide a vision dataset of 60 object-hand interactions. Heinrich et al (2018Heinrich et al ( , 2020 recorded the EMIL dataset on embodied multi-modal interaction for language learning. The dataset focuses on low-level crossmodal perception during the environmental interactions from a body-rational perspective.…”

Section: Datasetsmentioning

confidence: 99%

“…In the experimental setup, NICO is seated at a table with appropriate dimensions for a child-sized robot, as depicted in Figure 7. This experimental setup was initially introduced by Kerzel and Wermter (2017b) and subsequently adapted for various studies related to visuomotor learning and crossmodal object interaction (e.g., Eppe et al, 2017;Kerzel et al, , 2019bHeinrich et al, 2018Heinrich et al, , 2020. Therefore, the experimental setup recreates a realistic neurorobotic learning scenario.…”

Section: Experimental Setup and Processmentioning

confidence: 99%

Teaching NICO How to Grasp: An Empirical Study on Crossmodal Social Interaction as a Key Factor for Robots Learning From Humans

Kerzel¹,

Pekarek-Rosin²,

Strahl³

et al. 2020

Front. Neurorobot.

Self Cite

View full text Add to dashboard Cite

To overcome novel challenges in complex domestic environments, humanoid robots can learn from human teachers. We propose that the capability for social interaction should be a key factor in this teaching process and benefits both the subjective experience of the human user and the learning process itself. To support our hypothesis, we present a Human-Robot Interaction study on human-assisted visuomotor learning with the robot NICO, the Neuro-Inspired COmpanion, a child-sized humanoid. NICO is a flexible, social platform with sensing and manipulation abilities. We give a detailed description of NICO's design and a comprehensive overview of studies that use or evaluate NICO. To engage in social interaction, NICO can express stylized facial expressions and utter speech via an Embodied Dialogue System. NICO is characterized in particular by combining these social interaction capabilities with the abilities for human-like object manipulation and crossmodal perception. In the presented study, NICO acquires visuomotor grasping skills by interacting with its environment. In contrast to methods like motor babbling, the learning process is, in part, supported by a human teacher. To begin the learning process, an object is placed into NICO's hand, and if this object is accidentally dropped, the human assistant has to recover it. The study is conducted with 24 participants with little or no prior experience with robots. In the robot-guided experimental condition, assistance is actively requested by NICO via the Embodied Dialogue System. In the human-guided condition, instructions are given by a human experimenter, while NICO remains silent. Evaluation using established questionnaires like Godspeed, Mind Perception, and Uncanny Valley Indices, along with a structured interview and video analysis of the interaction, show that the robot's active requests for assistance foster the participant's engagement and benefit the learning process. This result supports the hypothesis that the ability for social interaction is a key factor for companion robots that learn with the help of non-expert teachers, as these robots become capable of communicating active requests or questions that are vital to their learning process. We also show how the design of NICO both enables and is driven by this approach.

show abstract

Crossmodal Language Grounding in an Embodied Neurocognitive Model

Cited by 24 publications

References 63 publications

The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

Learning Actions From Natural Language Instructions Using an ON-World Embodied Cognitive Architecture

Teaching NICO How to Grasp: An Empirical Study on Crossmodal Social Interaction as a Key Factor for Robots Learning From Humans

Contact Info

Product

Resources

About