Visual speech information for face recognition

Abstract

Two experiments test whether isolated visible speech movements can be used for face matching. Visible speech information was isolated with a point-light methodology. Participants were asked to match articulating point-light faces to a fully illuminated articulating face in an XAB task. The first experiment tested single-frame static face stimuli as a control. The results revealed that the participants were significantly better at matching the dynamic face stimuli than the static ones. Experiment 2 tested whether the observed dynamic advantage was based on the movement itself or on the fact that the dynamic stimuli consisted of many more static and ordered frames. For this purpose, frame rate was reduced, and the frames were shown in a random order, a correct order with incorrect relative timing, or a correct order with correct relative timing. The results revealedbetter matching performance with the correctly ordered and timed frame stimuli, suggesting that matches were based on the actual movement itself. These findings suggest that speaker-specific visible articulatory style can provide information for face matching.

Hearing a face: Cross-modal speaker matching using isolated visible speech

“…Analogous observations and arguments have been discussed in the visual speech literature (Rosenblum et al, 2002). In research modeled on the sine wave speech work, we found evidence that isolated visible articulatory information can be informative about speakers.…”

“…Research has shown that visual speech information can be recovered from these images (Rosenblum, Johnson, & Saldaña, 1996) and that they are treated as "true" visual speech stimuli in being automatically integrated with auditory speech (McGurk & MacDonald, 1976;. Our more recent research has shown that the isolated visible speech information contained in point-light stimuli can also inform about speaker identity (Rosenblum, Smith, & Niehus, 2006;Rosenblum et al, 2002). Despite the fact that these stimuli do not contain the facial feature and configuration information assumed necessary for recognition, point-light speech can be used for face recognition in both matching (Rosenblum et al, 2002) and identification contexts (Rosenblum et al, 2006; see also Bruce & Valentine, 1988).…”

“…Our more recent research has shown that the isolated visible speech information contained in point-light stimuli can also inform about speaker identity (Rosenblum, Smith, & Niehus, 2006;Rosenblum et al, 2002). Despite the fact that these stimuli do not contain the facial feature and configuration information assumed necessary for recognition, point-light speech can be used for face recognition in both matching (Rosenblum et al, 2002) and identification contexts (Rosenblum et al, 2006; see also Bruce & Valentine, 1988). These findings are consistent with other reports showing the salience of dynamic information for face recognition (Christie & Bruce, 1998;Knappmeyer, Thornton, & Bülthoff, 2003;Knight & Johnston, 1997;Lander, Christie, & Bruce, 1999).…”

“…For example, whether speech is p perceived through auditory, visual (lipread), or audiovisual means, familiarity with a speaker facilitates recognition (e.g., Nygaard & Pisoni, 1998;Schweinberger & Soukup, 1998;Sheffert & Olson, 2004;Yakel, Rosenblum, & Fortier, 2000). Moreover, there is evidence that, within both modalities, speech and speaker perception can make use of common informational dimensions (e.g., Rosenblum et al, 2002).…”

Learning to recognize talkers from natural, sinewave, and reversed speech samples.

“…Previous studies have shown that point-light displays of a talker's articulating face convey enough of the phonetic segmental grain to distinguish many phonemes and words and that this sensory stream is readily integrated with auditory speech (see Rosenblum & Saldaña, 1998, for a review). Moreover, these displays also permit the recognition of a familiar talker's face in silence (Rosenblum, Yakel, Baseer, & Panchal, 2002). Rosenblum et al have argued that subjects were able to recognize familiar faces by using idiolectal properties conveyed visually.…”