Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience

Abstract

In face-to-face communication, humans are faced with multiple layers of discontinuous multimodal signals, such as head, face, hand gestures, speech and non-speech sounds, which need to be interpreted as coherent and unified communicative actions. This implies a fundamental computational challenge: optimally binding only signals belonging to the same communicative action while segregating signals that are not connected by the communicative content. How do we achieve such an extraordinary feat, reliably, and efficiently? To address this question, we need to further move the study of human communication beyond speech-centred perspectives and promote a multimodal approach combined with interdisciplinary cooperation. Accordingly, we seek to reconcile two explanatory frameworks recently proposed in psycholinguistics and sensory neuroscience into a neurocognitive model of multimodal face-to-face communication. First, we introduce a psycholinguistic framework that characterises face-to-face communication at three parallel processing levels: multiplex signals, multimodal gestalts and multilevel predictions. Second, we consider the recent proposal of a lateral neural visual pathway specifically dedicated to the dynamic aspects of social perception and reconceive it from a multimodal perspective (“lateral processing pathway”). Third, we reconcile the two frameworks into a neurocognitive model that proposes how multiplex signals, multimodal gestalts, and multilevel predictions may be implemented along the lateral processing pathway. Finally, we advocate a multimodal and multidisciplinary research approach, combining state-of-the-art imaging techniques, computational modelling and artificial intelligence for future empirical testing of our model.

Progressive neural engagement within the IFG-pMTG circuit as gesture and speech entropy and MI advances

“…Contemporary perspectives on semantic processing portray it as a sequence of quantitatively functional mental states defined by a specific parser 3 , unified by statistical regularities among multiple sensory inputs 4 through hierarchical prediction and multimodal interactions [5][6][7][8][9] . Hence, proposals suggest that the coherent semantic representation emerges from statistical learning mechanisms within these 'convergence zones' or 'semantic hub' [10][11][12] , potentially functioning in a graded manner 12,13 .…”

WITHDRAWN: Progressive neural engagement within the IFG-pMTG circuit as gesture and speech entropy and MI advances

“…Contemporary perspectives on semantic processing portray it as a sequence of quantitatively functional mental states defined by a specific parser 3 , unified by statistical regularities among multiple sensory inputs 4 through hierarchical prediction and multimodal interactions [5][6][7][8][9] .…”

Interaction between neuroscience and happiness: assessment from Artificial Intelligence advances

“…In 2019, an artificial intelligence model was used to create a Happiness Index, and the results of the study coincided with several indicators established in the World Happiness Report related to freedom, income, trust, generosity, social support, and healthy life expectancy. (14) (15) In addition, other AI studies have looked at facial features to identify a person's feelings in certain situations. (11) (16) (126)(127) AI initiatives have emerged to solve various human problems such as poverty, human rights, or education using facial recognition, Convolutional or Recurrent Neural Networks, Computer Vision, and Natural Language Processing, among others.…”

“…Despite the complexity of happiness, neuroscience has been studying the brain processes and neural bases associated with positive emotions, and other aspects that can be applied in different areas of medicine are being checked with the support of AI. (12) As for the leading institutions in scientific production, the first three places are occupied by CNRS Centre National de la Recherche Scientifique (15), University College London (13) and Harvard Medical School (11) (Table 5).…”