Visual cortex entrains to sign language

Abstract

Despite immense variability across languages, people can learn to understand any human language, spoken or signed. What neural mechanisms allow people to comprehend language across sensory modalities? When people listen to speech, electrophysiological oscillations in auditory cortex entrain to slow (<8 Hz) fluctuations in the acoustic envelope. Entrainment to the speech envelope may reflect mechanisms specialized for auditory perception. Alternatively, flexible entrainment may be a general-purpose cortical mechanism that optimizes sensitivity to rhythmic information regardless of modality. Here, we test these proposals by examining cortical coherence to visual information in sign language. First, we develop a metric to quantify visual change over time. We find quasiperiodic fluctuations in sign language, characterized by lower frequencies than fluctuations in speech. Next, we test for entrainment of neural oscillations to visual change in sign language, using electroencephalography (EEG) in fluent speakers of American Sign Language (ASL) as they watch videos in ASL. We find significant cortical entrainment to visual oscillations in sign language <5 Hz, peaking at ∼1 Hz. Coherence to sign is strongest over occipital and parietal cortex, in contrast to speech, where coherence is strongest over the auditory cortex. Nonsigners also show coherence to sign language, but entrainment at frontal sites is reduced relative to fluent signers. These results demonstrate that flexible cortical entrainment to language does not depend on neural processes that are specific to auditory speech perception. Low-frequency oscillatory entrainment may reflect a general cortical mechanism that maximizes sensitivity to informational peaks in time-varying signals.sign language | cortical entrainment | oscillations | EEG L anguages differ dramatically from one another, yet people can learn to understand any natural language. What neural mechanisms allow humans to understand the vast diversity of languages and to distinguish linguistic signal from noise? One mechanism that has been implicated in language comprehension is neural entrainment to the volume envelope of speech. The volume envelope of speech fluctuates at low frequencies (< 8 Hz), decreasing at boundaries between syllables, words, and phrases. When people listen to speech, neural oscillations in the delta (1-4 Hz) and theta (4-8 Hz) bands become entrained to these fluctuations in volume (1-4).Entrainment to the volume envelope may represent an active neural mechanism to boost perceptual sensitivity to rhythmic stimuli (2, 5-7). Although entrainment is partly driven by bottom-up features of the stimulus (8-10), it also depends on top-down signals to auditory cortex from other brain areas. Auditory entrainment is strengthened when people see congruent visual and auditory information (11,12) and is modulated by attention (13) and by top-down signals from frontal cortex (4, 14).Cortical entrainment is proposed to perform a key role in speech comprehension, such as segmenting out sylla...

The left lateral occipital cortex exhibits decreased thickness in children with sensorineural hearing loss

“…Prior studies have demonstrated that non‐auditory sensory information such as visual and tactile inputs activate the auditory cortex in SNHL (Lomber et al, 2010; Sadato et al, 2005), suggesting substantial plasticity of the auditory cortex during brain development. It may be possible that the auditory cortex in SNHL patients receive atypical inputs from other brain regions such as visuo‐spatial information of sign language (Nishimura et al, 1999; Brookshire et al, 2017) to compensate for the loss of auditory signals.…”

“…It may be possible that the auditory cortex in SNHL patients receive atypical inputs from other brain regions such as visuo-spatial information of sign language (Nishimura et al, 1999;Brookshire et al, 2017) to compensate for the loss of auditory signals.…”

Pouw, Wassenburg, Hostetter, de Koning, &amp; Paas. Does Gesture Strengthen Sensorimotor Knowledge of Objects? The Case of the Size-Weight Illusion

“…We obtained the rate of movement (velocity) of gestures using a Frame Differencing Method (FDM; current sampling rate 25 frames per second). FDM utilizes an algorithm that computes the number of pixels that change from frame to frame from a video recording (using Python code made publicly available by Brookshire et al, 2017). This method provides an indication of gross movement through time-and is reliable compared to other methods such as Polhemus or Kinect (see Romero et al, 2017)-, which can be used as an estimate of velocity of hand-gesture movements.…”

“…We examined how often gestures were produced that closely mirrored the lifting motion involved in the actual task (e.g., lifting with two hands, rather than one), to see if producing such congruent gestures would be associated with a larger or smaller illusion. We also used a Frame Differencing Method (FDM; Brookshire, Lu, Nusbaum, Goldin-Meadow, & Cassasanto, 2017;Romero et al, 2017) to measure the velocity of two-handed lifting gestures to explore whether participants who report that the objects are heavier would move their hands more slowly as they gestured about lifting them. If gestures reflect such sensorimotor knowledge in their kinematics, this would provide strong evidence that gestures are based in sensorimotor know-how.…”

Time-resolved discrimination of audio-visual emotion expressions

“…In order to exclude the possibility that the general increase in performance at the increasing duration of the stimulation segments might be due to a simple increase in conveyed energy throughout the unfolding of the facial and vocal expressions, we investigated potential correlations between change in energy in our stimuli and change in the performance of the participants. The difference in root mean square (RMS) was measured across gates in auditory files, while the instantaneous visual change (IVC, defined in Brookshire et al 2017) was measured across frames of the visual stimuli for two separate subparts of the face, the eyes and the mouth. The differences in RMS across gates showed no positive correlation with the subjects' performance in the discrimination of vocal expressions across gates.…”