Multisensory training can promote or impede visual perceptual learning of speech stimuli: visual-tactile vs. visual-auditory training

Abstract

In a series of studies we have been investigating how multisensory training affects unisensory perceptual learning with speech stimuli. Previously, we reported that audiovisual (AV) training with speech stimuli can promote auditory-only (AO) perceptual learning in normal-hearing adults but can impede learning in congenitally deaf adults with late-acquired cochlear implants. Here, impeder and promoter effects were sought in normal-hearing adults who participated in lipreading training. In Experiment 1, visual-only (VO) training on paired associations between CVCVC nonsense word videos and nonsense pictures demonstrated that VO words could be learned to a high level of accuracy even by poor lipreaders. In Experiment 2, visual-auditory (VA) training in the same paradigm but with the addition of synchronous vocoded acoustic speech impeded VO learning of the stimuli in the paired-associates paradigm. In Experiment 3, the vocoded AO stimuli were shown to be less informative than the VO speech. Experiment 4 combined vibrotactile speech stimuli with the visual stimuli during training. Vibrotactile stimuli were shown to promote visual perceptual learning. In Experiment 5, no-training controls were used to show that training with visual speech carried over to consonant identification of untrained CVCVC stimuli but not to lipreading words in sentences. Across this and previous studies, multisensory training effects depended on the functional relationship between pathways engaged during training. Two principles are proposed to account for stimulus effects: (1) Stimuli presented to the trainee’s primary perceptual pathway will impede learning by a lower-rank pathway. (2) Stimuli presented to the trainee’s lower rank perceptual pathway will promote learning by a higher-rank pathway. The mechanisms supporting these principles are discussed in light of multisensory reverse hierarchy theory (RHT).

Cross-Modal Interaction Between Auditory and Visual Input Impacts Memory Retrieval

“…These include audio-visual illusions in which visual inputs bias the perceived location (e.g., the Ventriloquist Effect; Choe et al, 1975 ; see Vroomen and De Gelder, 2004 ) or identity (e.g., the McGurk Effect; McGurk and MacDonald, 1976 ; see Tiippana, 2014 ) of auditory input, as well as those characterizing effects of auditory signals on visual perception (e.g., the Double-Flash Illusion; Shams et al, 2000 ; see Keil, 2020 ). In fact, even short-term training with audio-visual stimuli can influence unimodal processing within the auditory system (e.g., Hazan et al, 2005 ; Song et al, 2008 ; Moradi et al, 2017 ), as well as the visual system (e.g., Eberhardt et al, 2014 ; Setti et al, 2014 ).…”

Learning Spoken Words via the Ears and Eyes: Evidence from 30-Month-Old Children

“…But the audio-visual experience was short and there was no evidence of cross-modal recognition upon visual learning. Besides, the literature concurrently reports that audio-visual experience with a word may either facilitate or hinder auditory and visual recognition of this word when learning word-object associations in adults ( Bernstein et al, 2013 ; Eberhardt et al, 2014 ). It is therefore unclear as to whether and how the multisensory information available during pre-familiarization influences lexical learning at 30 months and, if so, whether this effect changes over the course of development.…”

Speech-to-touch sensory substitution: a 10-decibel improvement in speech-in-noise understanding after a short training

“…unisensory) performance. A similar idea of a multisensory training boosting unisensory auditory and visual speech perception, was shown by Bernstein and colleagues [20] and by Eberhard and colleagues [25], respectively, although the applied language tasks were more basic than repeating whole sentences. At the same time, with future hard-of-hearing candidates for cochlear implantation, we believe that both unisensory tactile and multisensory audio-tactile training can be applied using our set-up, with the aim to "prepare" the auditory cortex for future processing of its natural sensory input [22,67].…”

“…Especially solutions combining multisensory inputs are appealing, as increasingly more experimental studies [17,18,19,20] and conceptual works point to the superiority of multisensory over unisensory training for learning and sensory recovery [21,22,23,24,25]. For improved understanding of auditory signals that are either degraded or presented in suboptimal conditions, multisensory training regimes that complement audition with vision have been found successful [20,25], including for rehabilitation of patients with hearing aids or cochlear implants, by adding speech reading or sign cues [1,6]. Interestingly, several recent works showed bene ts of adding tactile stimulation to improve degraded auditory speech comprehension, including our own ndings [26,27,28].…”