Degradation of Labial Information Modifies Audiovisual Speech Perception in Cochlear-Implanted Children

Huyse, Aurélie; Berthommier, Frédéric; Leybaert, Jacqueline

doi:10.1097/aud.0b013e3182670993

Cited by 28 publications

(49 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From that basis, we introduced a weighted fuzzy-logical model of perception, WFLMP, in which fusion would also involve specific weights controlling the role of each modality in the fusion process. This led to various implementation of the WFLMP, in which weights depend on the subject's individual characteristics (Schwartz, 2010;Huyse et al, 2013), attentional processes , or degradation of the auditory or visual input (Heckmann et al, 2002;Huyse et al, 2013).…”

Section: B Non-automaticity Of the Fusion Processmentioning

confidence: 99%

Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect

Nahorna

Berthommier

Schwartz

2015

The Journal of the Acoustical Society of America

Self Cite

View full text Add to dashboard Cite

While audiovisual interactions in speech perception have long been considered as automatic, recent data suggest that this is not the case. In a previous study, Nahorna et al. [(2012). J. Acoust. Soc. Am. 132, 1061-1077] showed that the McGurk effect is reduced by a previous incoherent audiovisual context. This was interpreted as showing the existence of an audiovisual binding stage controlling the fusion process. Incoherence would produce unbinding and decrease the weight of the visual input in fusion. The present paper explores the audiovisual binding system to characterize its dynamics. A first experiment assesses the dynamics of unbinding, and shows that it is rapid: An incoherent context less than 0.5 s long (typically one syllable) suffices to produce a maximal reduction in the McGurk effect. A second experiment tests the rebinding process, by presenting a short period of either coherent material or silence after the incoherent unbinding context. Coherence provides rebinding, with a recovery of the McGurk effect, while silence provides no rebinding and hence freezes the unbinding process. These experiments are interpreted in the framework of an audiovisual speech scene analysis process assessing the perceptual organization of an audiovisual speech input before decision takes place at a higher processing stage.

show abstract

Section: B Non-automaticity Of the Fusion Processmentioning

confidence: 99%

Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect

Nahorna

Berthommier

Schwartz

2015

The Journal of the Acoustical Society of America

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, the deaf children received their implants by at least three years of age, and most had used their implant for quite a few years, so auditory perception would be expected to be of higher rank in those children than in the deaf adults we studied who had received their implants late. Nevertheless, the Huyse et al (2013) study did show that perceptual weighting can be affected by adjusting visual clarity.…”

Section: Discussionmentioning

confidence: 96%

“…For example, Huyse et al (2013) showed that by reducing the clarity of visual speech in stimulus blocks comprising VO, AO, and audiovisual speech that speech perception was weighted more strongly towards the auditory stimuli in children with cochlear implants and ones with normal hearing. The weighting towards auditory speech is not surprising in normal-hearing children.…”

Section: Discussionmentioning

confidence: 99%

“…When visual stimuli were available to them, they may have relied on their lipreading ability rather than using the concordance between visual and auditory speech to learn the patterns in the auditory stimuli. In addition, they may have relied on their ability to integrate their relatively poor auditory representations with visual speech representations to achieve better multisensory perception (Giraud et al, 2001; Moody-Antonio et al, 2005; Rouger et al, 2008; Huyse et al, 2013). Either perceptual strategy would be expected to reduce auditory perceptual learning.…”

Section: Experiments 2: Lipreading Training With Va Vs Vo Stimulimentioning

confidence: 99%

“…Several studies have demonstrated that audiovisual (AV) training can promote perceptual learning of degraded auditory-only (AO) speech beyond training with AO stimuli (Pilling and Thomas, 2011; Wayne and Johnsrude, 2012; Bernstein et al, 2013, 2014; Huyse et al, 2013). Audiovisual training has also been shown to promote learning a phonemic contrast in a second language (Hardison, 2003; Hazan et al, 2006).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multisensory training can promote or impede visual perceptual learning of speech stimuli: visual-tactile vs. visual-auditory training

Eberhardt

Auer

Bernstein

2014

Front. Hum. Neurosci.

View full text Add to dashboard Cite

In a series of studies we have been investigating how multisensory training affects unisensory perceptual learning with speech stimuli. Previously, we reported that audiovisual (AV) training with speech stimuli can promote auditory-only (AO) perceptual learning in normal-hearing adults but can impede learning in congenitally deaf adults with late-acquired cochlear implants. Here, impeder and promoter effects were sought in normal-hearing adults who participated in lipreading training. In Experiment 1, visual-only (VO) training on paired associations between CVCVC nonsense word videos and nonsense pictures demonstrated that VO words could be learned to a high level of accuracy even by poor lipreaders. In Experiment 2, visual-auditory (VA) training in the same paradigm but with the addition of synchronous vocoded acoustic speech impeded VO learning of the stimuli in the paired-associates paradigm. In Experiment 3, the vocoded AO stimuli were shown to be less informative than the VO speech. Experiment 4 combined vibrotactile speech stimuli with the visual stimuli during training. Vibrotactile stimuli were shown to promote visual perceptual learning. In Experiment 5, no-training controls were used to show that training with visual speech carried over to consonant identification of untrained CVCVC stimuli but not to lipreading words in sentences. Across this and previous studies, multisensory training effects depended on the functional relationship between pathways engaged during training. Two principles are proposed to account for stimulus effects: (1) Stimuli presented to the trainee’s primary perceptual pathway will impede learning by a lower-rank pathway. (2) Stimuli presented to the trainee’s lower rank perceptual pathway will promote learning by a higher-rank pathway. The mechanisms supporting these principles are discussed in light of multisensory reverse hierarchy theory (RHT).

show abstract

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

Opoku-Baah

Schoenhaut

Vassall

et al. 2021

JARO

View full text Add to dashboard Cite

In a naturalistic environment, auditory cues are often accompanied by information from other senses, which can be redundant with or complementary to the auditory information. Although the multisensory interactions derived from this combination of information and that shape auditory function are seen across all sensory modalities, our greatest body of knowledge to date centers on how vision influences audition. In this review, we attempt to capture the state of our understanding at this point in time regarding this topic. Following a general introduction, the review is divided into 5 sections. In the first section, we review the psychophysical evidence in humans regarding vision’s influence in audition, making the distinction between vision’s ability to enhance versus alter auditory performance and perception. Three examples are then described that serve to highlight vision’s ability to modulate auditory processes: spatial ventriloquism, cross-modal dynamic capture, and the McGurk effect. The final part of this section discusses models that have been built based on available psychophysical data and that seek to provide greater mechanistic insights into how vision can impact audition. The second section reviews the extant neuroimaging and far-field imaging work on this topic, with a strong emphasis on the roles of feedforward and feedback processes, on imaging insights into the causal nature of audiovisual interactions, and on the limitations of current imaging-based approaches. These limitations point to a greater need for machine-learning-based decoding approaches toward understanding how auditory representations are shaped by vision. The third section reviews the wealth of neuroanatomical and neurophysiological data from animal models that highlights audiovisual interactions at the neuronal and circuit level in both subcortical and cortical structures. It also speaks to the functional significance of audiovisual interactions for two critically important facets of auditory perception—scene analysis and communication. The fourth section presents current evidence for alterations in audiovisual processes in three clinical conditions: autism, schizophrenia, and sensorineural hearing loss. These changes in audiovisual interactions are postulated to have cascading effects on higher-order domains of dysfunction in these conditions. The final section highlights ongoing work seeking to leverage our knowledge of audiovisual interactions to develop better remediation approaches to these sensory-based disorders, founded in concepts of perceptual plasticity in which vision has been shown to have the capacity to facilitate auditory learning.

show abstract

Degradation of Labial Information Modifies Audiovisual Speech Perception in Cochlear-Implanted Children

Cited by 28 publications

References 39 publications

Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect

Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect

Multisensory training can promote or impede visual perceptual learning of speech stimuli: visual-tactile vs. visual-auditory training

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

Contact Info

Product

Resources

About