Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

Benssassi, Esma Mansouri; Ye, Juan

doi:10.1109/taffc.2021.3106254

Cited by 8 publications

(5 citation statements)

References 76 publications

(78 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these probability models are rarely applied to real data. On the other hand, researchers have achieved advanced performance in tasks like multimodal emotion recognition [14], while they do not provide a probabilistic interpretation of these models.…”

Section: Discussionmentioning

confidence: 99%

“…For instance, while the model proposed in this paper was derived and numerically simulated for only two sensory modalities, its framework can be extended to multiple sensory modalities. Furthermore, research suggests that the effectiveness of multisensory integration may be related to the synchrony of neural spike firing [1,4,14]. This paper only considered fully synchronous visual and auditory stimuli, and investigation on asynchronous multisensory stimuli is needed in the future.…”

Section: Discussionmentioning

confidence: 99%

“…Benssassi and colleagues investigated the influence of neural synchrony on multisensory integration in emotion recognition. The model was demonstrated performance comparable to state-of-the-art multimodal fusion techniques such as deep learning fusion [14].…”

Section: Introductionmentioning

confidence: 86%

“…Research into multisensory integration is an important subfield of neuroscience, as it helps determine the relationships between different sensory signals and enhances the detection and recognition of external events. [13,14]. In artificial intelligence, intelligent robots need to accurately locate objects by integrating image data and sensor data.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Probabilistic modeling and numerical simulation of neural circuits for multisensory integration

Xu,

Liu

2023

HSET

View full text Add to dashboard Cite

People in real life receive stimulus information through various senses, and the process by which the brain integrates this information is called multisensory integration. Multisensory integration is an important branch of neuroscience, and the research on its neural mechanism holds significant application value to the development of artificial intelligence such as designing intelligent robots. Researches suggests that the brain likely employs Bayesian rules to integrate information and make judgments. In machine learning, neural networks based on Spike-Timing-Dependent Plasticity (STDP) have shown promising results in multimodal emotion recognition. In this paper, we model a neural network based on STDP, try to explain spike events using a probabilistic model, and unify network output with Bayesian calculation. This paper uses numerical simulation to verify the performance of the proposed network in multisensory classification problems. The results show that multisensory integration can improve classification accuracy and is better than the popular supervised learning method.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 86%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Probabilistic modeling and numerical simulation of neural circuits for multisensory integration

Xu,

Liu

2023

HSET

View full text Add to dashboard Cite

show abstract

“…The Enhancement model (Benssassi and Ye 2023) is inspired by the enhancement theory for multisensory recognition, where visual information significantly influences auditory cortex activity in the human brain (Molholm et al 2002;Jessen and Kotz 2013). In this model, the auditory features extraction layer receives inputs not only from the auditory input layer but also from the visual layer.…”

Section: Related Workmentioning

confidence: 99%

ND-MRM: Neuronal Diversity Inspired Multisensory Recognition Model

Wang,

Fan,

Jia

et al. 2024

AAAI

View full text Add to dashboard Cite

Cross-sensory interaction is a key aspect for multisensory recognition. Without cross-sensory interaction, artificial neural networks show inferior performance in multisensory recognition. On the contrary, the human brain has an inherently remarkable ability in multisensory recognition, which stems from the diverse neurons that exhibit distinct responses to sensory inputs, especially the multisensory neurons with multisensory responses hence enabling cross-sensory interaction. Based on this neuronal diversity, we propose a Neuronal Diversity inspired Multisensory Recognition Model (ND-MRM), which, similar to the brain, comprises unisensory neurons and multisensory neurons. To reflect the different responses characteristics of diverse neurons in the brain, special connection constraints are innovatively designed to regulate the features transmission in the ND-MRM. Leveraging this novel concept of neuronal diversity, our model is biologically plausible, enabling more effective recognition of multisensory information. To validate the performance of the proposed ND-MRM, we employ a multisensory emotion recognition task as a case study. The results demonstrate that our model surpasses state-of-the-art brain-inspired baselines on two datasets, proving the potential of brain-inspired methods for advancing multisensory interaction and recognition.

show abstract

Processing Physiological Sensor Data in Near Real-Time as Social Signals for Their Use on Social Virtual Reality Platforms

2022

View full text Add to dashboard Cite

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

Cited by 8 publications

References 76 publications

Probabilistic modeling and numerical simulation of neural circuits for multisensory integration

Probabilistic modeling and numerical simulation of neural circuits for multisensory integration

ND-MRM: Neuronal Diversity Inspired Multisensory Recognition Model

Processing Physiological Sensor Data in Near Real-Time as Social Signals for Their Use on Social Virtual Reality Platforms

Contact Info

Product

Resources

About