Multimodal FeedForward Self-organizing Maps

Paplinski, Andrew Peter; Gustafsson, Lennart

doi:10.1007/11596448_11

Cited by 10 publications

(11 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The feedback from the bimodal area to the auditory area should thus cause activity there even in the absence of auditory stimuli, provided a visual stimulus is present. In later modeling work (Gustafsson, Jantvik, & Papliński, 2007), an extended artificial neural network architecture with more versatile modules than those used previously (Papliński & Gustafsson, 2005Chou et al, 2007) was shown to replicate this property also, while still replicating the properties of earlier versions of the architecture.…”

Section: Introductionmentioning

confidence: 92%

A Self-Organized Artificial Neural Network Architecture for Sensory Integration with Applications to Letter-Phoneme Integration

2011

Self Cite

View full text Add to dashboard Cite

The multimodal self-organizing network (MMSON), an artificial neural network architecture carrying out sensory integration, is presented here. The architecture is designed using neurophysiological findings and imaging studies that pertain to sensory integration and consists of interconnected lattices of artificial neurons. In this artificial neural architecture, the degree of recognition of stimuli, that is, the perceived reliability of stimuli in the various subnetworks, is included in the computation. The MMSON's behavior is compared to aspects of brain function that deal with sensory integration. According to human behavioral studies, integration of signals from sensory receptors of different modalities enhances perception of objects and events and also reduces time to detection. In neocortex, integration takes place in bimodal and multimodal association areas and result, not only in feedback-mediated enhanced unimodal perception and shortened reaction time, but also in robust bimodal or multimodal percepts. Simulation data from the presented artificial neural network architecture show that it replicates these important psychological and neuroscientific characteristics of sensory integration.

show abstract

Section: Introductionmentioning

confidence: 92%

A Self-Organized Artificial Neural Network Architecture for Sensory Integration with Applications to Letter-Phoneme Integration

2011

Self Cite

View full text Add to dashboard Cite

show abstract

“…This association can then be used to influence the selforganization of monomodal maps [46]. The use of self-organizing maps to learn multimodal associations has been studied by several authors [88,68,35,71,84,40]. If these architectures are good at learning crossmodal associations, they suffer from the curse of dimensionality: projecting high dimensional data to two or three dimensions by preserving local topology (on which usually rely crossmodal associations) is difficult.…”

Section: Multimodal Fusionmentioning

confidence: 99%

Deep unsupervised network for multimodal perception, representation and classification

Droniou

Ivaldi

Sigaud

2015

Robotics and Autonomous Systems

View full text Add to dashboard Cite

In this paper, we tackle the problem of multimodal learning for autonomous robots. Autonomous robots interacting with humans in an evolving environment need the ability to acquire knowledge from their multiple perceptual channels in an unsupervised way. Most of the approaches in the literature exploit engineered methods to process each perceptual modality. In contrast, robots should be able to acquire their own features from the raw sensors, leveraging the information elicited by interaction with their environment: learning from their sensorimotor experience would result in a more efficient strategy in a life-long perspective. To this end, we propose an architecture based on deep networks, which is used by the humanoid robot iCub to learn a task from multiple perceptual modalities (proprioception, vision, audition). By structuring high-dimensional, multimodal information into a set of distinct sub-manifolds in a fully unsupervised way, it performs a substantial dimensionality reduction by providing both a symbolic representation of data and a fine discrimination between two similar stimuli. Moreover, the proposed network is able to exploit multimodal correlations to improve the representation of each modality alone.

show abstract

“…Kohonen Self-Organizing Maps (SOMs) are well-recognized and intensively researched tools for mapping multidimensional stimuli onto a low dimensionality (typically 2) neuronal lattice, for an introduction and a review, see [26]. In this paper we will employ a network of interconnected SOMs, referred to as Multimodal Self-Organizing Networks (MuSONs), see [23,24]. We consider first a feedforward Multimodal Self-Organizing Network (Mu-SON) as presented in the left part of Figure 1.…”

Section: The Multimodal Self-organizing Networkmentioning

confidence: 99%

“…Three-dimensional outputs from these maps, y lt and y ph , are combined together to form a six-dimensional stimulus for the higher-level bimodal map, SOM bm . The learning process takes place concurrently for each SOM, according to the well-known Kohonen learning law, see [23,24] The position of the winner can be determined from the network (map) postsynaptic activity, d(k) = W · x(k). As an example, in Figure 2, we show the post-synaptic activity of a trained bimodal map when the visual letter stimulus x lt (k) and the auditory phoneme stimulus x ph (k) representing letter/phonemeö is presented.…”

Section: The Multimodal Self-organizing Networkmentioning

confidence: 99%

“…In this paper, which is a direct continuation of work presented in [23,24], we model the processing of phonemes and letters both in sensory-specific areas and in sensory integration. The network we use for this study is a multimodal selforganizing network (MuSON) which consists of unimodal maps with phonetic and graphic inputs respectively, and an integrating bimodal map.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Feedback in Multimodal Self-organizing Networks Enhances Perception of Corrupted Stimuli

Paplinski

Gustafsson

2006

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. It is known from psychology and neuroscience that multimodal integration of sensory information enhances the perception of stimuli that are corrupted in one or more modalities. A prominent example of this is that auditory perception of speech is enhanced when speech is bimodal, i.e. when it also has a visual modality. The function of the cortical network processing speech in auditory and visual cortices and in multimodal association areas, is modeled with a Multimodal Self-Organizing Network (MuSON), consisting of several Kohonen Self-Organizing Maps (SOM) with both feedforward and feedback connections. Simulations with heavily corrupted phonemes and uncorrupted letters as inputs to the MuSON demonstrate a strongly enhanced auditory perception. This is explained by feedback from the bimodal area into the auditory stream, as in cortical processing.

show abstract

Multimodal FeedForward Self-organizing Maps

Cited by 10 publications

References 18 publications

A Self-Organized Artificial Neural Network Architecture for Sensory Integration with Applications to Letter-Phoneme Integration

A Self-Organized Artificial Neural Network Architecture for Sensory Integration with Applications to Letter-Phoneme Integration

Deep unsupervised network for multimodal perception, representation and classification

Feedback in Multimodal Self-organizing Networks Enhances Perception of Corrupted Stimuli

Contact Info

Product

Resources

About