Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching

Abstract

We introduce a seemingly impossible task: given only an audio clip of someone speaking, decide which of two face images is the speaker. In this paper we study this, and a number of related cross-modal tasks, aimed at answering the question: how much can we infer from the voice about the face and vice versa?We study this task "in the wild", employing the datasets that are now publicly available for face recognition from static images (VGGFace) and speaker identification from audio (VoxCeleb). These provide training and testing scenarios for both static and dynamic testing of cross-modal matching. We make the following contributions: (i) we introduce CNN architectures for both binary and multi-way cross-modal face and audio matching; (ii) we compare dynamic testing (where video information is available, but the audio is not from the same video) with static testing (where only a single still image is available); and (iii) we use human testing as a baseline to calibrate the difficulty of the task. We show that a CNN can indeed be trained to solve this task in both the static and dynamic scenarios, and is even well above chance on 10-way classification of the face given the voice. The CNN matches human performance on easy examples (e.g. different gender across faces) but exceeds human performance on more challenging examples (e.g. faces with the same gender, age and nationality) 1 .

Deep Latent Space Learning for Cross-Modal Mapping of Audio and Visual Signals

“…Recently, a dataset tailored towards audio-visual biometrics was introduced [6], [29] to aid the learning of audio and visual information and thus obtaining a joint representation. Many works have been focused towards speaker recognition and matching from audio and visual signals [5], [8]. Although these works effectively capture cross modal embeddings, however they require either separate networks for each modality and/or require pair selection during training to effectively penalize the negative pairs.…”

“…However, recently VoxCeleb dataset [6] has been introduced which comprises of a collection of video and audio recordings of a large number of celebrities. Previous works in literature [7], [5], [8] have modeled the problem of cross modal matching by employing separate networks for multiple modalities in either triplet network fashion or subnetwork. Separate networks in triplet fashion may help with modularity given few modalities (two in this case) at input, but it is important to take into account the possibility of multiple input streams (text, image, voice, etc).…”

“…Nagrani et. al [5] performed experiments in static and dynamic settings. A five stream dynamic-fusion architecture 1 requires five subnetworks to account for this fusion.…”

“…Our network performs under no restrictions in terms of triplet selection. We perform series of experiments inspired from [5], [9]. Furthermore, we perform two additional experiments to establish the robustness of our methodology.…”

An Extreme Value Theory Model of Cross-Modal Sensory Information Integration in Modulation of Vertebrate Visual System Functions

“…Beyond the merits of cultivating a better understanding of the operation of cross-modal sensory information integration in vertebrates, there is the possibility that an accurate computational model for this phenomenon could translate into a general algorithm for pattern recognition tasks in computer science. A direct application of this 2/20 method lies in the development of novel information fusion algorithms that leverage inputs from multiple sensory modalities, i.e, vision and audition [34]. Another practical application is the invention of innovative sensors capable of detecting changes in the environment and then re-configuring on the fly to change operational parameters and power consumption requirements.…”