Language experience influences audiovisual speech integration in unimodal and bimodal bilingual infants

Abstract

Infants as young as 2 months can integrate audio and visual aspects of speech articulation. A shift of attention from the eyes towards the mouth of talking faces occurs around 6 months of age in monolingual infants. However, it is unknown whether this pattern of attention during audiovisual speech processing is influenced by speech and language experience in infancy. The present study investigated this question by analysing audiovisual speech processing in three groups of 4- to 8-month-old infants who differed in their language experience: monolinguals, unimodal bilinguals (infants exposed to two or more spoken languages) and bimodal bilinguals (hearing infants with Deaf mothers). Eye-tracking was used to study patterns of face scanning while infants were viewing faces articulating syllables with congruent, incongruent and silent auditory tracks. Monolinguals and unimodal bilinguals increased their attention to the mouth of talking faces between 4 and 8 months, while bimodal bilinguals did not show any age difference in their scanning patterns. Moreover, older (6.6 to 8 months), but not younger, monolinguals (4 to 6.5 months) showed increased visual attention to the mouth of faces articulating audiovisually incongruent rather than congruent faces, indicating surprise or novelty. In contrast, no audiovisual congruency effect was found in unimodal or bimodal bilinguals. Results suggest that speech and language experience influences audiovisual integration in infancy. Specifically, reduced or more variable experience of audiovisual speech from the primary caregiver may lead to less sensitivity to the integration of audio and visual cues of speech articulation.

Impact of Language Experience on Attention to Faces in Infancy: Evidence From Unimodal and Bimodal Bilingual Infants

“…However, the magnitude of these effects did not differ from those of monolinguals. It was previously observed that bimodal bilinguals demonstrated increased looking time to talking faces in comparison to monolinguals ( Mercure et al, 2018 ). However, the present results suggest that this effect does not translate to static faces within a complex array.…”

“…For this reason, the presence of a static face without spoken or sign language production may not lead to increased attention to faces in bimodal bilinguals as it does for unimodal bilinguals. However, if the face begins to produce speech, increased attention to the face would be observed in bimodal bilinguals as a strategy to process a language modality in which the infant has less experience than monolinguals ( Mercure et al, 2018 ).…”

“…Bimodal bilingualism and the experience of communication in the visual modality with a Deaf mother do not appear to impact attention to unfamiliar still faces. Increased attention to faces for bimodal bilinguals compared to monolinguals may be restricted to talking faces in this group ( Mercure et al, 2018 ). Our data suggest that there are complex interactions in the development of face processing and language learning in the context of social communication in infancy.…”

“…In contrast, 12-month-old bilinguals show a preference for the mouth of faces talking in both native and non-native languages. Moreover, bilingual 8-month-olds are better than monolingual infants of the same age at distinguishing two different languages when silently articulated ( Weikum et al, 2007 ; Sebastián-Gallés et al, 2012 ), and bilingual infants from 4 to 8 months tend to spend more time looking at talking faces than monolinguals ( Mercure et al, 2018 ). Increased attention to the mouth was also observed for faces displaying non-linguistic emotional movements in 8-month-old bilinguals compared to monolinguals ( Ayneto and Sebastian-Galles, 2017 ), suggesting that bilingualism influences face scanning patterns beyond the context of speech processing.…”

“…Like unimodal bilinguals, bimodal bilinguals may achieve this more complex task by increasing their attention to faces. Congruent with this idea, using eye-tracking, we have previously reported that bimodal bilingual infants spend longer looking at talking faces than monolingual infants ( Mercure et al, 2018 ). Moreover, because infants with Deaf mothers often experience visual forms of communication, visual attention is key to their communicative experience with their mother and other Deaf people in their environment.…”