Matching heard and seen speech: An ERP study of audiovisual word recognition

Abstract

Seeing articulatory gestures while listening to speech-in-noise (SIN) significantly improves speech understanding. However, the degree of this improvement varies greatly among individuals. We examined a relationship between two distinct stages of visual articulatory processing and the SIN accuracy by combining a cross-modal repetition priming task with ERP recordings. Participants first heard a word referring to a common object (e.g., pumpkin) and then decided whether the subsequently presented visual silent articulation matched the word they had just heard. Incongruent articulations elicited a significantly enhanced N400, indicative of a mismatch detection at the pre-lexical level. Congruent articulations elicited a significantly larger LPC, indexing articulatory word recognition. Only the N400 difference between incongruent and congruent trials was significantly correlated with individuals’ SIN accuracy improvement in the presence of the talker’s face.

The Role of Audiovisual Speech in the Early Stages of Lexical Processing as Revealed by the ERP Word Repetition Effect

“…In a cross‐modal repetition paradigm, Kaganovich, Schumaker, and Rowland () observed that the N400 is sensitive to the match between auditory and visual speech cues. In their study, participants received auditory‐only words followed by visual‐only words that either matched the initial auditory‐only words or not.…”

“…Kaganovich et al. () asked their participants to determine whether the visual‐only word matched the word they had heard just before. This is different from the implicit processing of ongoing speech without any mismatch as performed by our participants.…”

“…However, contrary to our hypothesis, we did not observe any difference between auditory-only and audiovisual modalities in sentence context. In a cross-modal repetition paradigm, Kaganovich, Schumaker, and Rowland (2016) observed that the N400 is sensitive to the match between auditory and visual speech cues. In their study, participants received auditory-only words followed by visual-only words that either matched the initial auditoryonly words or not.…”

“…Contrary to their finding, we did not observe any effect of visual modality on the N400, perhaps owing to differences in the experimental procedure. Kaganovich et al (2016) asked their participants to determine whether the visual-only word matched the word they had heard just before. This is different from the implicit processing of ongoing speech without any mismatch as performed by our participants.…”

Electrophysiological Indices of Audiovisual Speech Perception: Beyond the McGurk Effect and Speech in Noise

“…Given that deficits in AV processing have been associated with complex neurodevelopmental disorders such as autism spectrum disorders (ASD) and specific language impairment (SLI; e.g., Bebko et al, 2006;Foss-Feig et al, 2010;Iarocci et al, 2010;Irwin et al, 2011;Kaganovich et al, 2014Kaganovich et al, , 2016Smith and Bennetto, 2007), better understanding of the neural bases for typical and atypical AV speech perception will be useful for identifying potential biomarkers that could indicate communication disorders.…”