Hearing a face: Cross-modal speaker matching using isolated visible speech

Abstract

An experiment was performed to test whether cross-modal speaker matches could be made using isolated visible speech movement information. Visible speech movements were isolated using a pointlight technique. In five conditions, subjects were asked to match a voice to one of two (unimodal) speaking point-light faces on the basis of speaker identity. Two of these conditions were designed to maintain the idiosyncratic speech dynamics of the speakers, whereas three of the conditions deleted or distorted the dynamics in various ways. Some of these conditions also equated video frames across dynamically correct and distorted movements. The results revealed generally better matching performance in the conditions that maintained the correct speech dynamics than in those conditions that did not, despite containing exactly the same video frames. The results suggest that visible speech movements themselves can support cross-modal speaker matching.

Alignment to visual speech information

“…In fact, this information allows perceivers to match heard speech to lipread speech on the basis of talker identity (e.g., Kamachi, Hill, Lander, & Vatikiotis-Bateson, 2003;Lachs & Pisoni, 2004a, 2004b, 2004cRosenblum et al, 2006). This suggests that speaking style can be perceived across modalities.…”

“…In supporting cross-modal matches, this information might best be construed as amodal or modality neutral. In fact, the notion of amodal talker-specific information has been used to explain the cross-modal talker matching findings described earlier (Kamachi et al, 2003;Lachs & Pisoni, 2004a, 2004b, 2004cRosenblum et al, 2006). The authors of those reports suggest that crossmodal matching could be based on the extraction of common idiolectic information available across modalities.…”

“…An acoustical analysis also revealed that the VOTs of the subjects' shadowed tokens were significantly longer than the VOTs of their baseline these stimuli, talkers can be recognized in both matching and identification contexts (Rosenblum, Niehus, & Smith, 2007;Rosenblum et al, 2002). Point-light research has also shown that a talker's isolated speech movements can be matched to their voice at better-than-chance levels (e.g., Lachs & Pisoni, 2004c;Rosenblum, Smith, Nichols, Hale, & Lee, 2006). These findings suggest that observers are sensitive to the articulatory style of a talker as it is reflected in both auditory and visual modalities.…”

Introducing crossmodal biometrics: Person identification from distinct audio & visual streams

“…Recent studies have investigated these phenomena from the viewpoint of human perception and psychophysics [10][11] [12][13] [14]. In these studies, human observers were asked to match an audio recording of an unknown voice X to two video (visual-only) recordings of two unknown speakers, A and B, one of which is X, and vice versa, under a variety of experimental conditions.…”

“…In these studies, human observers were asked to match an audio recording of an unknown voice X to two video (visual-only) recordings of two unknown speakers, A and B, one of which is X, and vice versa, under a variety of experimental conditions. Lachs et al [11], Rosenblum et al [14] and Kamachi et al [10] reported human observers correctly matching X to A or B around 65% of the times compared to the chance value of 50%. This was shown to be statistically significant given the number of independent test cases considered.…”