Multisensory processing is of vital importance for survival in the external world. Brain circuits can both integrate and separate visual and vestibular senses to infer self-motion and the motion of other objects. However, it is largely debated how multisensory brain regions process such multisensory information and whether they follow the Bayesian strategy in this process. Here, we combined macaque physiological recordings in the dorsal medial superior temporal area (MST-d) with modeling of synaptically coupled multilayer continuous attractor neural networks (CANNs) to study the underlying neuronal circuit mechanisms. In contrast to previous theoretical studies that focused on unisensory direction preference, our analysis showed that synaptic coupling induced cooperation and competition in the multisensory circuit and caused single MST-d neurons to switch between sensory integration or separation modes based on the fixed-criterion causal strategy, which is determined by the synaptic coupling strength. Furthermore, the prior of sensory reliability was represented by pooling diversified criteria at the MST-d population level, and the Bayesian strategy was achieved in downstream neurons whose causal inference flexibly changed with the prior. The CANN model also showed that synaptic input balance is the dynamic origin of neuronal direction preference formation and further explained the misalignment between direction preference and inference observed in previous studies. This work provides a computational framework for a new brain-inspired algorithm underlying multisensory computation.

The adult brain demonstrates remarkable multisensory plasticity by dynamically recalibrating itself based on information from multiple sensory sources. After a systematic visual–vestibular heading offset is experienced, the unisensory perceptual estimates for subsequently presented stimuli are shifted toward each other (in opposite directions) to reduce the conflict. The neural substrate of this recalibration is unknown. Here, we recorded single-neuron activity from the dorsal medial superior temporal (MSTd), parietoinsular vestibular cortex (PIVC), and ventral intraparietal (VIP) areas in three male rhesus macaques during this visual–vestibular recalibration. Both visual and vestibular neuronal tuning curves in MSTd shifted – each according to their respective cues’ perceptual shifts. Tuning of vestibular neurons in PIVC also shifted in the same direction as vestibular perceptual shifts (cells were not robustly tuned to the visual stimuli). By contrast, VIP neurons demonstrated a unique phenomenon: both vestibular and visual tuning shifted in accordance with vestibular perceptual shifts. Such that, visual tuning shifted, surprisingly, contrary to visual perceptual shifts. Therefore, while unsupervised recalibration (to reduce cue conflict) occurs in early multisensory cortices, higher-level VIP reflects only a global shift, in vestibular space.

The adult brain demonstrates remarkable multisensory plasticity by dynamically recalibrating information from multiple sensory sources. When a systematic visual-vestibular heading offset is experienced, the unisensory perceptual estimates recalibrate toward each other (in opposite directions) to reduce the conflict. The neural substrate of this recalibration is unknown. Here, we recorded single-neuron activity from the dorsal medial superior temporal (MSTd), parieto-insular vestibular cortex (PIVC), and ventral intraparietal (VIP) areas in three male rhesus macaques during visual-vestibular recalibration. Both visual and vestibular tuning in MSTd recalibrated - each according to their respective cues’ perceptual shifts. Vestibular tuning in PIVC also recalibrated together with corresponding perceptual shifts (cells were not visually tuned). By contrast, VIP neurons demonstrated a unique phenomenon: both vestibular and visual tuning recalibrated according to vestibular perceptual shifts. Such that, visual tuning shifted, surprisingly, contrary to visual perceptual shifts. Therefore, while unsupervised recalibration (to reduce cue conflict) occurs in early multisensory cortices, higher-level VIP reflects only a global shift, in vestibular space.

No abstract

The dorsomedial posterior parietal cortex is part of a higher-cognition network implicated in elaborate processes underpinning memory formation, recollection, episodes reconstruction, and temporal processing. Neural coding for complex episodic processing is however largely undocumented. Here we revealed a set of neural codes of 'neuroethogram' in the primate parietal cortex. Analyzing neural responses in macaque dmPPC to naturalistic videos, we discovered several groups of neurons that are sensitive to categories of ethogram-items and to low-level sensory features. The amount of information coded within these multiplex representations in turn increases our trained classifier decodability for different video-types. We further discovered that the processing of category and feature information by these neurons is sustained by accumulation of temporal information over a long timescale, corroborating its role at the apex of the cortical hierarchy of temporal receptive windows. Taken altogether, these neural findings explain how dorsomedial PPC weaves fabrics of ongoing experiences together in real-time and realize a multiplex representation of an organism's past. The high dimensionality of neural representations should motivate us to shift the focus of attention from pure selectivity neurons to mixed selectivity neurons, especially in increasingly complex naturalistic task designs.