Modeling development of natural multi-sensory integration using neural self-organisation and probabilistic population codes

Bauer, Johannes; Dávila-Chacón, Jorge; Wermter, Stefan

doi:10.1080/09540091.2014.971224

Cited by 10 publications

(9 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As we have demonstrated in the previous sections, it reproduces the convergence of primary and secondary sensory information in that brain region, the SC's topographic organization, and its unsupervised adaptation to stimulus statistics. Also, we have previously [7] shown that it can reproduce the spatial principle and the principle of inverse effectiveness, as well as maximum likelihood estimator (MLE)-like behavioral multisensory integration which is presumably caused by the neural processes in the SC. The SC is one multisensory region in the brain, whose input-output behavior is particularly well understood, and knowledge we glean about it can inform research on others [53].…”

Section: Discussionmentioning

confidence: 97%

“…The output of the network is a population-coded approximation of a probability density function (PDF) for the position of a stimulus. We have shown [5,7] that this model reproduces important aspects of natural MSI, namely the spatial principle, the principle of inverse effectiveness, and so-called optimal multisensory integration [39,30,51,2].…”

Section: Introductionmentioning

confidence: 98%

“…Focussing on sensory input, we have recently presented a model of learning MSI in the superior colliculus (SC) which is based on the self-organizing map (SOM) algorithm [6,7]. In that model, neurons learn the firing statistics of each of their input neurons and use these statistics to approximately compute and encode the probability of a stimulus being in their receptive field.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Attention modeled as information in learning multisensory integration

2015

Self Cite

View full text Add to dashboard Cite

Top-down cognitive processes affect the way bottom-up cross-sensory stimuli are integrated. In this paper, we therefore extend a successful previous neural network model of learning multisensory integration in the superior colliculus (SC) by top-down, attentional input and train it on different classes of cross-modal stimuli. The network not only learns to integrate cross-modal stimuli, but the model also reproduces neurons specializing in different combinations of modalities as well as behavioral and neurophysiological phenomena associated with spatial and feature-based attention. Importantly, we do not provide the model with any information about which input neurons are sensory and which are attentional. If the basic mechanisms of our model-self-organized learning of input statistics and divisive normalization-play a major role in the ontogenesis of the SC, then this work shows that these mechanisms suffice to explain a wide range of aspects both of bottom-up multisensory integration and the top-down influence on multisensory integration.

show abstract

Section: Discussionmentioning

confidence: 97%

Section: Introductionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Attention modeled as information in learning multisensory integration

2015

Self Cite

View full text Add to dashboard Cite

show abstract

“…The domain knowledge that is used later in life can be derived from the primitives that are encountered early in childhood, for example, in interactions between infants and parents, and is referred to as intermodal synchrony (Rohlfing and Nomikou 2014). As a further example, our own research shows that learning, which is based on crossmodal integration, like the integration of real sensory perception on low and on intermediate levels (as suggested for the superior colliculus in the brain), can enable both super-additivity and dominance of certain modalities based on the tasks (Bauer et al 2015).…”

Section: Stefan Wermter Sascha Griffiths and Stefan Heinrichmentioning

confidence: 90%

The architecture challenge: Future artificial-intelligence systems will require sophisticated architectures, and knowledge of the brain might guide their construction

Baldassarre¹,

Santucci²,

Cartoni³

et al. 2017

Behav Brain Sci

View full text Add to dashboard Cite

show abstract

“…For some objects and actions the data contains salient features in a certain modality, while for others, all modalities are necessary for disambiguation. This allows studying mechanisms on sensor fusion, superadditivity, and hierarchical composition in addition to embodied representation formation on the cortex-level (Bauer et al, 2015 ).…”

Section: Related Data Setsmentioning

confidence: 99%

Crossmodal Language Grounding in an Embodied Neurocognitive Model

Heinrich

Yao

Hinz³

et al. 2020

Front. Neurorobot.

Self Cite

View full text Add to dashboard Cite

Human infants are able to acquire natural language seemingly easily at an early age. Their language learning seems to occur simultaneously with learning other cognitive functions as well as with playful interactions with the environment and caregivers. From a neuroscientific perspective, natural language is embodied, grounded in most, if not all, sensory and sensorimotor modalities, and acquired by means of crossmodal integration. However, characterizing the underlying mechanisms in the brain is difficult and explaining the grounding of language in crossmodal perception and action remains challenging. In this paper, we present a neurocognitive model for language grounding which reflects bio-inspired mechanisms such as an implicit adaptation of timescales as well as end-to-end multimodal abstraction. It addresses developmental robotic interaction and extends its learning capabilities using larger-scale knowledge-based data. In our scenario, we utilize the humanoid robot NICO in obtaining the EMIL data collection, in which the cognitive robot interacts with objects in a children's playground environment while receiving linguistic labels from a caregiver. The model analysis shows that crossmodally integrated representations are sufficient for acquiring language merely from sensory input through interaction with objects in an environment. The representations self-organize hierarchically and embed temporal and spatial information through composition and decomposition. This model can also provide the basis for further crossmodal integration of perceptually grounded cognitive representations.

show abstract

Modeling development of natural multi-sensory integration using neural self-organisation and probabilistic population codes

Cited by 10 publications

References 40 publications

Attention modeled as information in learning multisensory integration

Attention modeled as information in learning multisensory integration

The architecture challenge: Future artificial-intelligence systems will require sophisticated architectures, and knowledge of the brain might guide their construction

Crossmodal Language Grounding in an Embodied Neurocognitive Model

Contact Info

Product

Resources

About