Lip movements entrain the observers’ low-frequency brain oscillations to facilitate speech intelligibility

Park, Hyojin; Kayser, Christoph; Thut, Gregor; Groß, Joachim

doi:10.7554/elife.14521

Cited by 152 publications

(279 citation statements)

References 62 publications

Supporting

Mentioning

227

Contrasting

Order By: Relevance

“…Entrainment to auditory speech is strongest over auditory cortex (2,8,11,35) and central frontal sites (14). By contrast, our results show that entrainment to sign language is strongest at occipital and parietal channels, consistent with greater parietal activation during sign compared with speech (53).…”

Section: Neural Mechanisms Of Language Comprehension Across Sensorycontrasting

confidence: 50%

“…Although these rhythmic preferences are not absolute [visual cortex also shows rhythmic oscillations in the delta/theta range (11,17,(34)(35)(36)(37)], differences between the sensory modalities are apparent when different frequency bands are directly compared. Auditory detection sensitivity depends on the phase of underlying delta-theta, but not alpha, oscillations (28).…”

Section: Discussionmentioning

confidence: 99%

“…Previous results are consistent with this interpretation as well. When participants watch videos of speech, entrainment emerges not only in auditory cortex, but also in visual cortex (11,35,36). Furthermore, cortical rhythms entrain to rhythms in music (42) and to other rhythmic stimuli in audition (29,30) and vision (19,43).…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Visual cortex entrains to sign language

Brookshire

Nusbaum

et al. 2017

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Despite immense variability across languages, people can learn to understand any human language, spoken or signed. What neural mechanisms allow people to comprehend language across sensory modalities? When people listen to speech, electrophysiological oscillations in auditory cortex entrain to slow (<8 Hz) fluctuations in the acoustic envelope. Entrainment to the speech envelope may reflect mechanisms specialized for auditory perception. Alternatively, flexible entrainment may be a general-purpose cortical mechanism that optimizes sensitivity to rhythmic information regardless of modality. Here, we test these proposals by examining cortical coherence to visual information in sign language. First, we develop a metric to quantify visual change over time. We find quasiperiodic fluctuations in sign language, characterized by lower frequencies than fluctuations in speech. Next, we test for entrainment of neural oscillations to visual change in sign language, using electroencephalography (EEG) in fluent speakers of American Sign Language (ASL) as they watch videos in ASL. We find significant cortical entrainment to visual oscillations in sign language <5 Hz, peaking at ∼1 Hz. Coherence to sign is strongest over occipital and parietal cortex, in contrast to speech, where coherence is strongest over the auditory cortex. Nonsigners also show coherence to sign language, but entrainment at frontal sites is reduced relative to fluent signers. These results demonstrate that flexible cortical entrainment to language does not depend on neural processes that are specific to auditory speech perception. Low-frequency oscillatory entrainment may reflect a general cortical mechanism that maximizes sensitivity to informational peaks in time-varying signals.sign language | cortical entrainment | oscillations | EEG L anguages differ dramatically from one another, yet people can learn to understand any natural language. What neural mechanisms allow humans to understand the vast diversity of languages and to distinguish linguistic signal from noise? One mechanism that has been implicated in language comprehension is neural entrainment to the volume envelope of speech. The volume envelope of speech fluctuates at low frequencies (< 8 Hz), decreasing at boundaries between syllables, words, and phrases. When people listen to speech, neural oscillations in the delta (1-4 Hz) and theta (4-8 Hz) bands become entrained to these fluctuations in volume (1-4).Entrainment to the volume envelope may represent an active neural mechanism to boost perceptual sensitivity to rhythmic stimuli (2, 5-7). Although entrainment is partly driven by bottom-up features of the stimulus (8-10), it also depends on top-down signals to auditory cortex from other brain areas. Auditory entrainment is strengthened when people see congruent visual and auditory information (11,12) and is modulated by attention (13) and by top-down signals from frontal cortex (4, 14).Cortical entrainment is proposed to perform a key role in speech comprehension, such as segmenting out sylla...

show abstract

Section: Neural Mechanisms Of Language Comprehension Across Sensorycontrasting

confidence: 50%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Visual cortex entrains to sign language

Brookshire

Nusbaum

et al. 2017

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Neural entrainment to continuous visual speech has been previously studied in the context of physical, low-level information through the use of the acoustic envelope (Crosse et al, 2015a) and lip movements (Park et al, 2016). Park et al (2016) showed that high-level visual regions as well as speech processing regions specifically entrained to the visual component of speech.…”

Section: Discussionmentioning

confidence: 99%

“…And fMRI research on AV speech perception has shown increases in connectivity strength between putatively multisensory regions and visual cortex when the visual modality is more reliable (Nath and Beauchamp, 2011). In terms of continuous visual speech processing, recent MEG work found extensive (bilateral) entrainment of visual cortex to visual speech (lip movements) when the visual signal was relevant for speech comprehension (Park et al, 2016). Importantly, this entrainment was restricted to a much smaller area in early visual cortex (left-lateralized) when the visual speech was irrelevant.…”

Section: Introductionmentioning

confidence: 99%

Visual Cortical Entrainment to Motion and Categorical Speech Features during Silent Lipreading

O’Sullivan

Crosse

Liberto

et al. 2017

Front. Hum. Neurosci.

View full text Add to dashboard Cite

Speech is a multisensory percept, comprising an auditory and visual component. While the content and processing pathways of audio speech have been well characterized, the visual component is less well understood. In this work, we expand current methodologies using system identification to introduce a framework that facilitates the study of visual speech in its natural, continuous form. Specifically, we use models based on the unheard acoustic envelope (E), the motion signal (M) and categorical visual speech features (V) to predict EEG activity during silent lipreading. Our results show that each of these models performs similarly at predicting EEG in visual regions and that respective combinations of the individual models (EV, MV, EM and EMV) provide an improved prediction of the neural activity over their constituent models. In comparing these different combinations, we find that the model incorporating all three types of features (EMV) outperforms the individual models, as well as both the EV and MV models, while it performs similarly to the EM model. Importantly, EM does not outperform EV and MV, which, considering the higher dimensionality of the V model, suggests that more data is needed to clarify this finding. Nevertheless, the performance of EMV, and comparisons of the subject performances for the three individual models, provides further evidence to suggest that visual regions are involved in both low-level processing of stimulus dynamics and categorical speech perception. This framework may prove useful for investigating modality-specific processing of visual speech under naturalistic conditions.

show abstract

Is student learning from a video lecture affected by whether the instructor wears a mask?

Leng,

Wang,

Mayer

2024

Applied Cognitive Psychology

View full text Add to dashboard Cite

This study examined whether having the instructor wear a mask during a video lecture affects learning. In Experiment 1, college students watched an instructional video on the formation of lightning, in which an instructor who either did or did not wear a mask as she stood next to slides and lectured. Learners' learning outcomes did not differ significantly, but learners spent significantly less time looking at the instructor's face when she was masked. In Experiment 2, using a 2 (the instructor wore a mask or not) × 2 (slides were displayed or not) between‐subject design, college students learned about the process of water cycle from instructional videos. There was a significant interaction in which adding slides improved learning outcomes with a masked instructor, but not with an unmasked instructor. Adding a mask lowered student ratings of social presence with the instructor. Practical and theoretical implications are discussed.

show abstract

Lip movements entrain the observers’ low-frequency brain oscillations to facilitate speech intelligibility

Cited by 152 publications

References 62 publications

Visual cortex entrains to sign language

Visual cortex entrains to sign language

Visual Cortical Entrainment to Motion and Categorical Speech Features during Silent Lipreading

Is student learning from a video lecture affected by whether the instructor wears a mask?

Contact Info

Product

Resources

About