Learning to Parse Grounded Language using Reservoir Computing

Abstract

Recently new models for language processing and learning using Reservoir Computing have been popular. However, these models are typically not grounded in sensorimotor systems and robots. In this paper, we develop a model of Reservoir Computing called Reservoir Parser (ResPars) for learning to parse Natural Language from grounded data coming from humanoid robots. Previous work showed that ResPars is able to do syntactic generalization over different sentences (surface structure) with the same meaning (deep structure). We argue that such ability is key to guide linguistic generalization in a grounded architecture. We show that ResPars is able to generalize on grounded compositional semantics by combining it with Incremental Recruitment Language (IRL). Additionally, we show that ResPars is able to learn to generalize on the same sentences, but not processed word by word, but as an unsegmented sequence of phonemes. This ability enables the architecture to not rely only on the words recognized by a speech recognizer, but to process the sub-word level directly. We additionally test the model's robustness to word error recognition.

ReservoirPy: An Efficient and User-Friendly Library to Design Echo State Networks

“…Thus, online learning proposes a lighter approach to train reservoir with less computational demand while still achieving compatible level of accuracy. More importantly perhaps, online incremental learning methods are crucial for computational neuroscience models [14] and developmental experiments in cognitive science (developmental psychology, robotics, ...) [6,11]. Current implementation of online learning in ReservoirPy is based on FORCE learning method [15], and resides in a separate class: ESNOnline.…”

“…A decade ago, some integrated libraries were available, like Oger 5 in Python language, or aureservoir [7] in C++. Several projects on ESNs can be found on Github 6 . However, there is currently no equivalent library to Oger.…”

“…Several members of our team and students already used it for different tasks and purposes (e.g. to build Computational Neuroscience models and Human-Robot Interaction modules [6,11,14]), it is now time to share it more extensively.…”

Editorial: Language and Robotics

“…Taking inspiration of how children acquire language can help to design the simplest mechanisms to deal with these challenges. Conversely, robotics can help modeling and test hypotheses about language acquisition and language grounding (Cangelosi and Schlesinger, 2015;Taniguchi et al, 2016Taniguchi et al, , 2018Hinaut and Spranger, 2019), in particular through cross-situational experiments (Taniguchi et al, 2017;Juven and Hinaut, 2020).…”

Language Acquisition with Echo State Networks: Towards Unsupervised Learning

“…Some psychological studies investigate the bases of interaction created between an infant and its caregiver [5], necessary to root verbal communication and the formation of abstract symbols [6]. Harnad stated the famous Symbol Grounding Problem [7] and shown the importance for a system manipulating symbol to "anchor" their meanings to "raw" perceptions: the so-called symbol grounding that several studies experimented in robots [8], [9]. From a developmental perspective, a bottom-up approach that let symbols emerge seems relatively feasible and more appropriate [10].…”