Sensorimotor input as a language generalisation tool: a neurorobotics model for generation and generalisation of noun-verb combinations with sensorimotor inputs

Abstract

The paper presents a neurorobotics cognitive model explaining the understanding and generalisation of nouns and verbs combinations when a vocal command consisting of a verb-noun sentence is provided to a humanoid robot. The dataset used for training was obtained from object manipulation tasks with a humanoid robot platform; it includes 9 motor actions and 9 objects placing placed in 6 different locations), which enables the robot to learn to handle real-world objects and actions. Based on the multiple time-scale recurrent neural networks, this study demonstrates its generalisation capability using a large data-set, with which the robot was able to generalise semantic representation of novel combinations of noun-verb sentences, and therefore produce the corresponding motor behaviours. This generalisation process is done via the grounding process: different objects are being interacted, and associated, with different motor behaviours, following a learning approach inspired by developmental language acquisition in infants. Further analyses of the learned network dynamics and representations also demonstrate how the generalisation is possible via the exploitation of this functional hierarchical recurrent network.

Encoding Longer-term Contextual Sensorimotor Information in a Predictive Coding Model

“…Thus, the whole network is able to work as a number of non-linear dynamic functions as a similar role of bifurcation. While the model is used to learn the temporal sequences such as the sensorimotor information of the robots, the model is able to represent different spatio-temporal embodiment scales of sensorimotor information, such as the language learning [27,25] and object features/movements [28]. Similar concept of multiple time-scales has also be applied in Gated Recurrent Units for automatically context extraction [29,30].…”

“…Compared with the τ values selected in MTRNN works (e.g. [26,28] 3 and Fig. 4 show the comparison between the samples of the original and the predicted images.…”

A Brief Review of Neural Networks Based Learning and Control and Their Applications for Robots

“…In terms of its hierarchical organization, it also allows this operation: with bidirectional information pathways, a low level perception representation can be expressed on a higher level, with a more complex receptive field, and vice versa low ⇔ high . This can be realized by the bidirectional deep architectures such as [112]. Conceptually, these operations can be achieved by extracting statistical regularity shown in Figure 7.…”

“…For instance, in the MTRNN network [112], the learning of each neuron follows the updating rule of classical firing rate models, in which the activity of a neuron is determined by the average firing rate of all the connected neurons. Additionally, the neuronal activity is also decaying over time following an updating rule of leaky integrator model.…”

“…Based on this architecture, two-layer RNN models were utilized to extract visual information [119] and to understand intentions [120] or emotion status [121] in social robotics; three-layer RNN models were used to integrate and understand multimodal information for a humanoid iCub robot [112,122]. Moreover, the predictive coding framework has been extended to variational Bayes predictive coding MTRNN, which can arbitrate between deterministic model and probabilistic model by setting a metaparameter [123].…”