Speech2Face: Learning the Face Behind a Voice

Abstract

How much can we infer about a person's looks from the way they speak? In this paper, we study the task of reconstructing a facial image of a person from a short audio recording of that person speaking. We design and train a deep neural network to perform this task using millions of natural Internet/YouTube videos of people speaking. During training, our model learns voice-face correlations that allow it to produce images that capture various physical attributes of the speakers such as age, gender and ethnicity. This is done in a self-supervised manner, by utilizing the natural co-occurrence of faces and speech in Internet videos, without the need to model attributes explicitly. We evaluate and numerically quantify how-and in what manner-our Speech2Face reconstructions, obtained directly from audio, resemble the true face images of the speakers.

Direct Speech-to-Image Translation

“…Hao et al [12] proposed a uniform framework using cycle constraint for the visual-audio mutual generation. Recently, Oh et al [8] presented a model for generating the face images from a voice using a pretrained face decoder, but the generated results are not sharp due to the concern of privacy. Duarte et al [14] generated sharp face images conditioned on the input speech segmentation using GANs.…”

“…Following this work, several text-to-image models [3], [4], [10] were proposed for the text-to-image task, based on the same teacher-student learning method to train the text encoder. Recently, teacher-student learning was used to generate the face behind a voice [13]. As a comparison, traditional audioto-image generation models [11] used a classifier as the feature extractor.…”

“…Although the class label of the speech description is accessible, there might be generalization problem when the trained model is tested on the new unseen data (new class respect to the training set). Inspired by the cross-modal generation models [3], [7], [13], [16], [23], in this work, we use teacher-student learning [30] to overcome this problem to some extent.…”

“…Text-to-image translation [3]- [5], [7] is a closely related topic to ours, which has been investigated for several years. In some text-to-image models, zero-shot learning based methods [7], [8] and generative adversarial networks (GANs) [9] have been used to extract features and synthesize realistic images, respectively. These models generalize better on the new testing classes by leveraging the teacher-student learning to train the text encoder [7], [10].…”

“…In addition to the text-to-image translation, several models for audio-toimage generation were also presented in recent years. Chen et al [11] and Hao et al [12] synthesized instrument images from different music inputs; Oh et al [13] and Amanda et al [14] reconstructed the human face images from input speech based on the positive correlations between a persons appearance and his voice, and both their frameworks contain a speech encoder and a face decoder. Different from these audio-to-image generation works, which model the acoustic or phonetic information mainly, our speech-to-image translation aims to model the linguistic information in the input speech and translate it into the images.…”