Does dynamic information about the speaker's face contribute to semantic speech processing? ERP evidence

Hernández-Gutiérrez, David; Rahman, Rasha Abdel; Martı́n-Loeches, Manuel; Muñoz, Francisco; Schacht, Annekathrin; Sommer, Werner

doi:10.1016/j.cortex.2018.03.031

Cited by 10 publications

(8 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Two previous studies have investigated the impact of mouth within the N400 time window. Hernández-Gutiérrez and colleagues did not find any N400 difference between audiovisual and audio-only speech [54]; while Brunellière and colleagues found an increase in N400 amplitude for more informative mouth movements [11]. Further research is necessary to clarify these discrepancies, however, our results suggest that mouth informativeness can affect processing in the N400 time window but only in combination with other cues in a multimodal context.…”

Section: Discussioncontrasting

confidence: 76%

“…However, while Brunellière and colleagues compared N400 of words starting with more or less informative mouth movements (/b/ v.s. /k/) and found that words with more informative mouth movements elicited more negative N400 [11], suggesting increased processing difficulty, Hernández-Gutiérrez and colleagues failed to find any N400 effect associated with mouth movements when comparing videos with dynamic facial and mouth movement and with a still image of the speaker [54].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension

et al. 2021

View full text Add to dashboard Cite

The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.

show abstract

Section: Discussioncontrasting

confidence: 76%

Section: Introductionmentioning

confidence: 99%

More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension

et al. 2021

View full text Add to dashboard Cite

show abstract

“…As both facial speech processing and phoneme/grapheme associations are fundamentally audiovisual processes (Francisco et al, 2018 ), and have been shown to partly share neural circuitry (Blomert, 2011 ), we were particularly interested in examining the possibility that the presence of an articulating mouth may enhance the quality of word encoding, by making phoneme/grapheme pairings clearer. Interestingly, the presence of articulation cues has been shown in prior experimental research to affect aspects of psycholinguistic processing, including facilitating upcoming word recognition (Hernández-Gutiérrez et al, 2018 ) and encoding during voice learning (Sheffert & Olson, 2004 ). For instance, in one study (Hernández-Gutiérrez et al, 2018 ), adults listened to short stories in which one target word was either expected from the story context or unexpected.…”

Section: Studymentioning

confidence: 99%

Facial speech processing in children with and without dyslexia

et al. 2021

View full text Add to dashboard Cite

What role does the presence of facial speech play for children with dyslexia? Current literature proposes two distinctive claims. One claim states that children with dyslexia make less use of visual information from the mouth during speech processing due to a deficit in recruitment of audiovisual areas. An opposing claim suggests that children with dyslexia are in fact reliant on such information in order to compensate for auditory/phonological impairments. The current paper aims at directly testing these contrasting hypotheses (here referred to as “mouth insensitivity” versus “mouth reliance”) in school-age children with and without dyslexia, matched on age and listening comprehension. Using eye tracking, in Study 1, we examined how children look at the mouth across conditions varying in speech processing demands. The results did not indicate significant group differences in looking at the mouth. However, correlation analyses suggest potentially important distinctions within the dyslexia group: those children with dyslexia who are better readers attended more to the mouth while presented with a person’s face in a phonologically demanding condition. In Study 2, we examined whether the presence of facial speech cues is functionally beneficial when a child is encoding written words. The results indicated lack of overall group differences on the task, although those with less severe reading problems in the dyslexia group were more accurate when reading words that were presented with articulatory facial speech cues. Collectively, our results suggest that children with dyslexia differ in their “mouth reliance” versus “mouth insensitivity,” a profile that seems to be related to the severity of their reading problems.

show abstract

“…Two previous studies have investigated the impact of mouth movements within the N400 time window. Hernández-Gutiérrez and colleagues did not find any N400 difference between audiovisual and audio-only speech 52 ;…”

Section: Prosody Gesture and Mouth Movements Contribution To Linguismentioning

confidence: 92%

“…Two electrophysiological studies, however, reported conflicting findings. While Brunellière and colleagues linked more informative mouth movements to more negative N400 amplitude 51 , generally indicating increased processing difficulty, Hernández-Gutiérrez and colleagues failed to find any N400 effect associated with mouth movements 52 .…”

Section: Introductionmentioning

confidence: 96%

Word predictability, prosody, gesture and mouth movements in face-to-face language comprehension

Zhang

Frassinelli

Tuomainen

et al. 2020

Preprint

View full text Add to dashboard Cite

The natural ecology of human language is face-to-face interaction, comprising cues, like cospeech gestures, mouth movements and prosody, tightly synchronized with speech. Yet, this rich multimodal context is usually stripped away in experimental studies as the dominant paradigm focuses on speech alone. We ask how these audio-visual cues impact brain activity during naturalistic language comprehension, how they are dynamically orchestrated and whether they are organized hierarchically. We quantify each cue in video-clips of a speaker and we used a well-established electroencephalographic marker of comprehension difficulties, an event-related potential, peaking around 400ms after word-onset. We found that multimodal cues always modulated brain activity in interaction with speech, that their impact dynamically changes with their informativeness and that there is a hierarchy: prosody shows the strongest effect followed by gestures and mouth movements. Thus, this study provides a first snapshot into how the brain dynamically weights audiovisual cues in real-world language comprehension. Electrophysiology of multimodal comprehension ! 3 Electrophysiology of multimodal comprehension ! 4 frame theories of natural language processing because if some multimodal cues (e.g., gesture or prosody) always contribute to processing, this would imply that our current speech-only focus is too narrow, if not misleading. Second, we need to understand the dynamics of online multimodal comprehension. In particular, to provide mechanistic accounts of language comprehension, it is necessary to establish how the weight of a certain cue dynamically changes depending upon the context (e.g., whether meaningful hand gestures are weighted more when prior linguistic context is less informative and/or when mouth movements are less informative). Finally, it is important to establish whether there is a stable hierarchical organization of cues (e.g., prior linguistic context may always be weighted more than gestures, which are in turn weighted more than mouth movements). Prosody, gesture and mouth movements as predictors of upcoming words: the state of the artAccentuation (i.e., prosodic stress characterized as higher pitch that makes words acoustically prominent) marks new information 10 . Many behavioural studies have revealed that comprehension is facilitated with appropriate accentuation (new information is accentuated, and old information de-accentuated 11,12 . Incongruence between the presence of prosodic accentuation and newness of information increases processing difficulty, inducing increased activation in left inferior frontal gyrus, interpreted as increased phonological and semantic processing difficulty 13 . In electrophysiological (EEG) studies, such mismatch elicits more negative N400 (an event-related-potential (ERP) peaking negatively 400ms after word presentation around central-parietal areas 14 , that has been argued to mark prediction in language comprehension 2 ) than appropriate accentuation 15-20 .Electrophysiology of multimod...

show abstract

Does dynamic information about the speaker's face contribute to semantic speech processing? ERP evidence

Cited by 10 publications

References 46 publications

More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension

More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension

Facial speech processing in children with and without dyslexia

Word predictability, prosody, gesture and mouth movements in face-to-face language comprehension

Contact Info

Product

Resources

About