Computational modeling of human multisensory spatial representation by a neural architecture

Abstract

Our brain constantly combines sensory information in unitary percept to build coherent representations of the environment. Even though this process could appear smooth, integrating sensory inputs from various sensory modalities must overcome several computational issues, such as recoding and statistical inferences problems. Following these assumptions, we developed a neural architecture replicating humans’ ability to use audiovisual spatial representations. We considered the well-known ventriloquist illusion as a benchmark to evaluate its phenomenological plausibility. Our model closely replicated human perceptual behavior, proving a truthful approximation of the brain’s ability to develop audiovisual spatial representations. Considering its ability to model audiovisual performance in a spatial localization task, we release our model in conjunction with the dataset we recorded for its validation. We believe it will be a powerful tool to model and better understand multisensory integration processes in experimental and rehabilitation environments.

Digital Twins for Supporting Ageing Well: Approaches in Current Research and Innovation in Europe and Japan

“…Research in modeling human behavior is being conducted, including various aspects of natural communication skills and their imitation. This involves not only auditory information for automatic speech recognition [28,29] and speaker analysis (also called Computational Paralinguistics [30], but also the use of visual information like multimodal face [31,32], hand gesturing [33], and body movements [34,35], as well as other sensor information like tactile, olfactory, and gustatory [36], as well as artificial neurons [37] and the whole cognition [38].…”