Neural Correlates of Temporal Complexity and Synchrony during Audiovisual Correspondence Detection

Abstract

Visual Abstract

Audio-visual combination of syllables involves time-sensitive dynamics following from fusion failure

“…Interestingly, this sensitivity did not translate into a misalignment sensation, confirming that this range of AV asynchrony is perceptually well tolerated 4 . These findings are consistent with previous studies 23,24,62 showing that the IFG is involved in implicit timing 22 , a sensitivity that allows listeners to predict word order from syntactic information 63 . Even though our brain is not aware of precise event temporality, the latter is essential to monitor e.g., articulatory and syntactic correctness, and to prepare for correct production.…”

“…Yet, the detection of shorter temporal AV asynchronies is possible and takes place in other brain regions, in particular in the dorsal premotor area and the inferior frontal gyrus [22][23][24][25] .…”

“…Although the LSTS responds better when auditory and visual speech are perfectly synchronous 18,19 , its activity can cope with large temporal discrepancies, reflecting a broad temporal window of integration in the order of the syllable length (up to ~260 ms) 20 . This large window of integration can even be pathologically stretched to about 1s in subjects suffering from autism spectrum disorder 21 .Yet, the detection of shorter temporal AV asynchronies is possible and takes place in other brain regions, in particular in the dorsal premotor area and the inferior frontal gyrus [22][23][24][25] .considering the temporal patterns in a 2 nd acoustic formant/lip aperture two-dimensional (2D) feature space is sufficient to qualitatively reproduce participants' behaviour for fused 13,29 but also combined responses 28 . Simulations indicated that fusion is possible, and even expected, when the physical features of the A and V stimulus, represented by the 2 nd formant and lip in the model, are located in the neighbourhood of an existing 2D syllable representation.…”

Audio-visual combination of syllables involves time-sensitive dynamics following from fusion failure

“…This large integration window can even be pathologically stretched to about 1 s in subjects suffering from autism spectrum disorder 27 . Yet, the detection of shorter temporal AV asynchronies is possible and takes place in other brain regions, in particular in the dorsal premotor area and the inferior frontal gyrus [28][29][30][31] . The STS and the IFG regions hence exhibit different functions in AV speech integration 32,33 , owing to their different temporal integration properties 34 .…”

“…On the other hand, the frontal cortex has been widely associated with cognitive control 36 and temporal information processing in both short-and long-term memory 37,38 . In the context of AV speech, the inferior frontal cortex is involved in monitoring AV temporal sequences 30 .…”