A Multimodal Connectionist Architecture for Unsupervised Grounding of Spatial Language

Vavrecka, Michal; Farkaš, Igor

doi:10.1007/s12559-013-9212-5

Cited by 13 publications

(9 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The development of associations between co-occurring stimuli for multimodal binding has been strongly supported by neurophysiological evidence [62,112]. Similar to References [44,113,114] and based on our experimental results, we argue that the co-occurrence of sensory inputs is a sufficient source of information to create robust multimodal representations with the use of associative links between unimodal representations that can be incrementally learned in an unsupervised fashion.…”

Section: A Universal Multimodal Association Model?supporting

confidence: 88%

Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning

2020

View full text Add to dashboard Cite

Cortical plasticity is one of the main features that enable our ability to learn and adapt in our environment. Indeed, the cerebral cortex self-organizes itself through structural and synaptic plasticity mechanisms that are very likely at the basis of an extremely interesting characteristic of the human brain development: the multimodal association. In spite of the diversity of the sensory modalities, like sight, sound and touch, the brain arrives at the same concepts (convergence). Moreover, biological observations show that one modality can activate the internal representation of another modality when both are correlated (divergence). In this work, we propose the Reentrant Self-Organizing Map (ReSOM), a brain-inspired neural system based on the reentry theory using Self-Organizing Maps and Hebbian-like learning. We propose and compare different computational methods for unsupervised learning and inference, then quantify the gain of the ReSOM in a multimodal classification task. The divergence mechanism is used to label one modality based on the other, while the convergence mechanism is used to improve the overall accuracy of the system. We perform our experiments on a constructed written/spoken digits database and a Dynamic Vision Sensor (DVS)/EletroMyoGraphy (EMG) hand gestures database. The proposed model is implemented on a cellular neuromorphic architecture that enables distributed computing with local connectivity. We show the gain of the so-called hardware plasticity induced by the ReSOM, where the system’s topology is not fixed by the user but learned along the system’s experience through self-organization.

show abstract

Section: A Universal Multimodal Association Model?supporting

confidence: 88%

Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning

2020

View full text Add to dashboard Cite

show abstract

“…This approach postulates the unsupervised role of linguistic labels that can affect categorisation during the acquisition process, which has also been supported by Taniguchi et al [42]. Vavrečka and Farkaš [46] recently introduced a multimodal architecture for grounding of spatial words using a biologically inspired approach (separate "what" and "where" visual subsystems) in which the visual scenes (two objects in 2D space in a spatial relation) are associated with their linguistic descriptions, hence leading to integration of modalities.…”

Section: Introductionmentioning

confidence: 70%

Mapping Language to Vision in a Real-World Robotic Scenario

Štěpánová

Klein

Cangelosi

et al. 2018

IEEE Trans. Cogn. Dev. Syst.

View full text Add to dashboard Cite

Language has evolved over centuries and was gradually enriched and improved. The question, how people find assignment between meanings and referents, still remains open. There is plenty of computational models based on statistical co-occurrence of meaning-reference pairs. Unfortunately, this mapping strategy shows poor performance in the environment with higher number of objects or noise. Therefore we propose a more robust noise-resistant algorithm. We tested the performance of this novel algorithm both with simulated and physical iCub robot. We developed a testing scenario consisting of objects with varying visual properties presented to the robot accompanied by utterances describing the given object. Our results suggest that the proposed mapping procedure is robust, resistant against noise and shows better performance than one-step mapping for all levels of noise in the linguistic input as well as slower performance degradation with increasing noise. Furthermore, it increases the clustering accuracy of both modalities.

show abstract

“…We have investigated some researches which apply these self-organizing incremental neural networks into multimodality integration. For example, Vavrecka and Farkas [32] presented a two-layer connectionist architecture based on SOMs to bind spatial location, shape and color of objects. Parisi et al [33]- [35] built a series of hierarchical GWR networks to fuse multimodal action representations.…”

Section: (B) the Process Mainly Involves Audiovisual Integration Andmentioning

confidence: 99%

An Autonomous Developmental Cognitive Architecture Based on Incremental Associative Neural Network With Dynamic Audiovisual Fusion

Huang

Song

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Developing cognition is difficult to achieve yet crucial for robots. Infants can gradually improve their cognition through parental guidance and self-exploration. However, conventional learning methods for robots often focus on a single modality and train a pre-defined model by large datasets in an offline way. In this paper, we propose a hierarchical autonomous cognitive architecture for robots to learn object concepts online by interacting with humans. Two pathways for audiovisual information are devised. Each pathway has three layers based on the self-organizing incremental neural networks. Visual features and names of objects are incrementally learned and self-organized in an unsupervised way in sample layers, respectively, in which we propose a dynamically adjustable similarity threshold strategy to allow the network itself to control cluster rather than using a pre-defined threshold. Two symbol layers abstract the cluster results from the corresponding sample layer to form concise symbols and transmit them to an associative layer. An associative relationship between two modalities can be built in real time by binding activated visual and auditory symbols simultaneously in the associative layer. In this layer, a top-down response strategy is proposed to let robots autonomously recall another associative modality, solve conflicting associative relationships, and adjust learned knowledge from the top down. The experimental results on two objects datasets and a real task show that our architecture is efficient to learn and associate object view and name in an online way. What is more, the robot can autonomously improve its cognitive level by utilizing its own experience without enquiring with humans. INDEX TERMS Cognitive development, concept online learning, self-organizing incremental neural network, object recognition, audiovisual integration.

show abstract

A Multimodal Connectionist Architecture for Unsupervised Grounding of Spatial Language

Cited by 13 publications

References 36 publications

Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning

Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning

Mapping Language to Vision in a Real-World Robotic Scenario

An Autonomous Developmental Cognitive Architecture Based on Incremental Associative Neural Network With Dynamic Audiovisual Fusion

Contact Info

Product

Resources

About