The use of visual information in non-native speech sound discrimination across the first year of life

Abstract

Infants are able to match seen and heard speech even in non-native languages, and familiarization to audiovisual speech appears to affect subsequent auditory-only discrimination of non-native speech sounds (Danielson et al., 2013; 2014). However, the robustness of these behaviors appears to change rapidly within the first year of life. In this current set of studies, conducted with six-, nine-, and 10-month-old English-learning infants, we examine the developmental trajectory of audiovisual speech perception of non-native speech sounds. In the first place, we show that the tendency to detect a mismatch between heard and seen speech sounds in a non-native language changes across this short period in development, in tandem with the trajectory of auditory perceptual narrowing (Werker & Tees, 1984; Kuhl et al., 1992; inter alia). Furthermore, we demonstrate that infants’ familiarization to matching and mismatching audiovisual speech affects their auditory speech perception differently at various ages. While six-month-old infants’ auditory speech perception appears to be malleable in the face of prior audiovisual familiarization, this tendency declines with age. The current set of studies is one of the first to utilize traditional looking-time measurements while also employing pupillometry as a correlate of infants’ acoustic change detection (Hochmann & Papeo, 2014).

Discriminating Non-native Vowels on the Basis of Multimodal, Auditory or Visual Information: Effects on Infants’ Looking Patterns and Discrimination

“…For instance, infants’ sensitivity to a non-native vowel contrast can be improved with a short training phase that paired these vowels consistently with two distinct visual objects, although this only held for infants who went on to have larger vocabularies at 18 months ( Ter Schure et al, in press ). Also, observing simultaneously visual articulations affects discriminability of a (native) consonant contrast ( Teinonen et al, 2008 ), and the congruence or incongruence between non-native sounds and visual articulations can even alter infants’ listening preferences ( Danielson et al, 2015 ). Although in our study we did not observe an interaction effect between modality condition and distribution, we find that infants discriminate the vowel contrast after training with two-peaked visual plus auditory distributions.…”

“…For example, 2-month-old infants notice a mismatch between speech sounds and a speaking face ( Bristow et al, 2009 ) and infants between 2 and 5 months are able to match auditory and visual speech cues ( Kuhl and Meltzoff, 1982 ; Patterson and Werker, 2003 ; Kushnerenko et al, 2008 ; Bristow et al, 2009 ). The type of audiovisual speech can also affect 6-month-olds’ listening preferences for tokens from a novel phonetic contrast: when speech sounds match with the visual information, infants prefer alternating tokens over repeated tokens, whereas when the speech sounds are incongruent with the visual information (i.e., point to different phonemes), they prefer repeated tokens over alternating tokens ( Danielson et al, 2015 ). Pons et al (2009) suggested that intersensory perception for non-native contrasts declines between 6 and 11 months: Spanish 6-month-olds are better than 11-month-olds at matching the non-native (English) [ba]∼[va] contrast to the corresponding visual articulations.…”