Brief Sensory Training Narrows the Temporal Binding Window and Enhances Long-Term Multimodal Speech Perception

Abstract

Our ability to integrate multiple sensory-based representations of our surrounding supplies us with a more holistic view of our world. There are many complex algorithms our nervous system uses to construct a coherent perception. An indicator to solve this ‘binding problem’ are the temporal characteristics with the specificity that environmental information has different propagation speeds (e.g., sound and electromagnetic waves) and sensory processing time and thus the temporal relationship of a stimulus pair derived from the same event must be flexibly adjusted by our brain. This tolerance can be conceptualized in the form of the cross-modal temporal binding window (TBW). Several studies showed the plasticity of the TBW and its importance concerning audio-visual illusions, synesthesia, as well as psychiatric disturbances. Using three audio-visual paradigms, we investigated the importance of length (short vs. long) as well as modality (uni- vs. multimodal) of a perceptual training aiming at reducing the TBW in a healthy population. We also investigated the influence of the TBW on speech intelligibility, where participants had to integrate auditory and visual speech information from a videotaped speaker. We showed that simple sensory trainings can change the TBW and are capable of optimizing speech perception at a very naturalistic level. While the training-length had no different effect on the malleability of the TBW, the multisensory trainings induced a significantly stronger narrowing of the TBW than their unisensory counterparts. Furthermore, a narrowing of the TBW was associated with a better performance in speech perception, meaning that participants showed a greater capacity for integrating informations from different sensory modalities in situations with one modality impaired. All effects persisted at least seven days. Our findings show the significance of multisensory temporal processing regarding ecologically valid measures and have important clinical implications for interventions that may be used to alleviate debilitating conditions (e.g., autism, schizophrenia), in which multisensory temporal function is shown to be impaired.

Narrowing of the Audiovisual Temporal Binding Window Due To Perceptual Training Is Specific to High Visual Intensity Stimuli

“…The reduction in the TBW when training was performed with bright stimuli is consistent with previous reports ( De Niear et al., 2016 ; Powers et al., 2009 , 2012 ; Zerr et al., 2019 ). Our analysis also shows that this training effect is consistent with a shift in AV criterion placement, supporting evidence that trial-by-trial feedback can induce criterion shifts ( Aberg & Herzog, 2012 ) which can lead to performance improvements in perceptual tasks ( Herzog et al., 2006 ).…”

“…Increases in TBW size have been associated with increased fall risk in older age ( Setti et al., 2011 ) and poorer performance on speech perception tasks ( Conrey & Pisoni, 2006 ; Stevenson et al., 2018 ). Training healthy individuals at an audiovisual SJ task led to lasting improvements in speech perception ( Zerr et al., 2019 ). Training with the specific aim to reduce the TBW has consequently been proposed as an intervention for those with multisensory deficiencies ( Foss-Feig et al., 2010 ; Powers et al., 2009 ; Setti et al., 2014 ; Stevenson et al., 2017 ; Wallace & Stevenson, 2014 ).…”

“…The specificity of perceptual learning increases with training ( Jeter et al., 2010 ), and, to be useful across a range of conditions, a careful balance between specificity and generalisation has to be struck. This consideration may also explain conflicting results, such as the generalisation of unimodal TOJ training to bimodal stimuli (e.g., Stevenson et al., 2013 vs. Zerr et al., 2019 ).…”

A Randomized Controlled Trial for Audiovisual Multisensory Perception in Autistic Youth

“…The substantial evidence for altered audiovisual temporal binding in autistic youth, as well as observed relations between TBWs for audiovisual speech and other domains of functioning in autistic youth, has engendered increasing interest in the possibility of training audiovisual integration in autistic youth (e.g., Bahrick & Todd 2012 ; Cascio et al, 2016 ; Feldman et al, 2018 ; Zhou et al, 2020 ). A number of training studies targeting audiovisual temporal binding have been conducted in non-autistic adults, and have been shown to narrow TBWs in a relatively short period of time (i.e., 3–5 sessions; De Niear et al, 2016 , 2018 ; McGovern et al, 2016 ; Powers et al, 2009 ; Setti et al, 2014 ; Sürig et al, 2018 ; Zerr et al, 2019 ). These training paradigms provide automated feedback after each trial of a computerized task wherein participants must make judgements about the synchrony or temporal order of audiovisual stimuli, such as flashes and beeps (e.g., Powers et al, 2009 ; Setti et al, 2014 ; Sürig et al, 2018 ) and audiovisual speech, such as a speaker saying the syllable “ba” (De Niear et al, 2018 ).…”

“…There are several limitations to the literature on perceptual trainings for audiovisual stimuli in non-autistic adults. First, the majority of these perceptual training studies have found no evidence for generalization to untrained multisensory tasks (De Niear et al, 2018 ; Powers et al, 2016 ; Setti et al, 2014 ) or limited evidence for generalization (i.e., training effects only observed at limited SOAs or conditions in untrained tasks; Zerr et al, 2019 ). To date, only Sürig et al, ( 2018 ) have found strong evidence for generalization.…”