Seeing a talking face matters: The relationship between cortical tracking of continuous auditory‐visual speech and gaze behaviour in infants, children and adults

Tan, S. H. Jessica; Kalashnikova, Marina; Liberto, Giovanni M. Di; Crosse, Michael J.; Burnham, Denis

doi:10.1016/j.neuroimage.2022.119217

Cited by 19 publications

(28 citation statements)

References 91 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, neural tracking of auditory‐only nursery rhymes in 10‐ and 14‐month‐olds predicts vocabulary size at 24 months (Menn et al., 2022). For speech stimuli, neural tracking is greater in both infants and adults when audiovisual information from the speaker's face is available (Tan et al., 2022). Neural tracking is also greater in sung than spoken sentences in adults listening to audio only stimuli (der Nederlanden et al., 2020; der Nederlanden et al., 2022).…”

Section: Discussionmentioning

confidence: 99%

Infant‐directed song potentiates infants’ selective attention to adults’ mouths over the first year of life

Alviar

Sahoo

Edwards

et al. 2023

Developmental Science

View full text Add to dashboard Cite

The mechanisms by which infant‐directed (ID) speech and song support language development in infancy are poorly understood, with most prior investigations focused on the auditory components of these signals. However, the visual components of ID communication are also of fundamental importance for language learning: over the first year of life, infants’ visual attention to caregivers’ faces during ID speech switches from a focus on the eyes to a focus on the mouth, which provides synchronous visual cues that support speech and language development. Caregivers’ facial displays during ID song are highly effective for sustaining infants’ attention. Here we investigate if ID song specifically enhances infants’ attention to caregivers’ mouths. 299 typically developing infants watched clips of female actors engaging them with ID song and speech longitudinally at six time points from 3 to 12 months of age while eye‐tracking data was collected. Infants’ mouth‐looking significantly increased over the first year of life with a significantly greater increase during ID song versus speech. This difference was early‐emerging (evident in the first 6 months of age) and sustained over the first year. Follow‐up analyses indicated specific properties inherent to ID song (e.g., slower tempo, reduced rhythmic variability) in part contribute to infants’ increased mouth‐looking, with effects increasing with age. The exaggerated and expressive facial features that naturally accompany ID song may make it a particularly effective context for modulating infants’ visual attention and supporting speech and language development in both typically developing infants and those with or at risk for communication challenges. A video abstract of this article can be viewed at https://youtu.be/SZ8xQW8h93A.Research Highlights Infants’ visual attention to adults’ mouths during infant‐directed speech has been found to support speech and language development. Infant‐directed (ID) song promotes mouth‐looking by infants to a greater extent than does ID speech across the first year of life. Features characteristic of ID song such as slower tempo, increased rhythmicity, increased audiovisual synchrony, and increased positive affect, all increase infants’ attention to the mouth. The effects of song on infants’ attention to the mouth are more prominent during the second half of the first year of life.

show abstract

Section: Discussionmentioning

confidence: 99%

Infant‐directed song potentiates infants’ selective attention to adults’ mouths over the first year of life

Alviar

Sahoo

Edwards

et al. 2023

Developmental Science

View full text Add to dashboard Cite

show abstract

“…Five‐month‐old infants segmented words from the continuous speech of a target speaker in a multi‐talker environment when they could see the target speaker's articulatory movements, but not when they were presented with a still picture of the speaker (Hollich et al., 2005). Another study found that 5‐month‐old infants’ and adults’ cortical tracking of speech was enhanced when speech was audio‐visual compared to audio‐only, whereas 4‐year‐old children did not show this audio‐visual speech benefit (Tan et al., 2022). Furthermore, studies suggested a developmental shift in infants’ looking behaviour at a talking face, with infants gradually shifting their attention to the speaker's mouth rather than to the eyes between 4 and 8 months, and returning to attending preferentially to the speaker's eyes by 12 months (Lewkowicz & Hansen‐Tift, 2012; Tenenbaum et al., 2015).…”

Section: Discussionmentioning

confidence: 99%

“…For the ERP analyses, we excluded trials in which infants attended to the screen for less than 25% of the duration of the familiarisation phase, similar to previous eye‐tracking and EEG studies that suggested similar exclusion criteria (e.g. 15% in LoBue et al., 2017; 20% in Taylor & Herbert, 2013; also see Tan et al., 2022). This means that if the infant attended to the screen for less than 25% of the video duration time in the familiarisation phase, the subsequent ERP test trials that were part of the same block were discarded.…”

Section: Methodsmentioning

confidence: 99%

Does the speaker's eye gaze facilitate infants’ word segmentation from continuous speech? An ERP study

Çetinçelik

Rowland

Snijders

2023

Developmental Science

View full text Add to dashboard Cite

The environment in which infants learn language is multimodal and rich with social cues. Yet, the effects of such cues, such as eye contact, on early speech perception have not been closely examined. This study assessed the role of ostensive speech, signalled through the speaker's eye gaze direction, on infants’ word segmentation abilities. A familiarisation‐then‐test paradigm was used while electroencephalography (EEG) was recorded. Ten‐month‐old Dutch‐learning infants were familiarised with audio‐visual stories in which a speaker recited four sentences with one repeated target word. The speaker addressed them either with direct or with averted gaze while speaking. In the test phase following each story, infants heard familiar and novel words presented via audio‐only. Infants’ familiarity with the words was assessed using event‐related potentials (ERPs). As predicted, infants showed a negative‐going ERP familiarity effect to the isolated familiarised words relative to the novel words over the left‐frontal region of interest during the test phase. While the word familiarity effect did not differ as a function of the speaker's gaze over the left‐frontal region of interest, there was also a (not predicted) positive‐going early ERP familiarity effect over right fronto‐central and central electrodes in the direct gaze condition only. This study provides electrophysiological evidence that infants can segment words from audio‐visual speech, regardless of the ostensiveness of the speaker's communication. However, the speaker's gaze direction seems to influence the processing of familiar words.Research Highlights We examined 10‐month‐old infants’ ERP word familiarity response using audio‐visual stories, in which a speaker addressed infants with direct or averted gaze while speaking. Ten‐month‐old infants can segment and recognise familiar words from audio‐visual speech, indicated by their negative‐going ERP response to familiar, relative to novel, words. This negative‐going ERP word familiarity effect was present for isolated words over left‐frontal electrodes regardless of whether the speaker offered eye contact while speaking. An additional positivity in response to familiar words was observed for direct gaze only, over right fronto‐central and central electrodes.

show abstract

“…Although evidence of the neural tracking mechanisms in typically developing children is limited, more recent studies have been able to show successful neural tracking of continuous natural speech in both infants (< 1 year) (Kalashnikova et al, 2018; Tan et al, 2022; Attaheri et al, 2022) and young children between 4 and 9 years old (Vander Ghinst et al, 2019; Ríos-López et al, 2020; Tan et al, 2022). For example, Ríos-López et al (2020) showed that speech-brain coupling already occurs at 4 years of age in the delta-band frequency range, but not at theta frequencies.…”

Section: Introductionmentioning

confidence: 99%

Delta-band neural envelope tracking predicts speech intelligibility in noise in preschoolers

Hirtum

Somers

Verschueren

et al. 2023

Preprint

View full text Add to dashboard Cite

Behavioral tests are currently the gold standard in measuring speech intelligibility. However, these tests can be difficult to administer in young children due to factors such as motivation, linguistic knowledge and cognitive skills. It has been shown that measures of neural envelope tracking can be used to predict speech intelligibility and overcome these issues. However, its potential as an objective measure for speech intelligibility in noise remains to be investigated in preschool children. Here, we evaluated neural envelope tracking as a function of signal-to-noise ratio (SNR) in 14 5-year-old children. We examined EEG responses to natural, continuous speech presented at different SNRs ranging from -8 (very difficult) to 8 dB SNR (very easy). As expected delta band (0.5-4 Hz) tracking increased with increasing stimulus SNR. However, this increase was not strictly monotonic as neural tracking reached a plateau between 0 and 4 dB SNR, similarly to the behavioral speech intelligibility outcomes. These findings indicate that neural tracking in the delta band remains stable, as long as the acoustical degradation of the speech signal does not reflect significant changes in speech intelligibility. Theta band tracking (4-8 Hz), on the other hand, was found to be drastically reduced and more easily affected by noise in children, making it less reliable as a measure of speech intelligibility. By contrast, neural envelope tracking in the delta band was directly associated with behavioral measures of speech intelligibility. This suggests that neural envelope tracking in the delta band is a valuable tool for evaluating speech-in-noise intelligibility in preschoolers, highlighting its potential as an objective measure of speech in difficult-to-test populations.

show abstract

Seeing a talking face matters: The relationship between cortical tracking of continuous auditory‐visual speech and gaze behaviour in infants, children and adults

Cited by 19 publications

References 91 publications

Infant‐directed song potentiates infants’ selective attention to adults’ mouths over the first year of life

Infant‐directed song potentiates infants’ selective attention to adults’ mouths over the first year of life

Does the speaker's eye gaze facilitate infants’ word segmentation from continuous speech? An ERP study

Delta-band neural envelope tracking predicts speech intelligibility in noise in preschoolers

Contact Info

Product

Resources

About