Talking heads synthesis from audio with deep neural networks

Abstract

Talking heads synthesis with expressions from speech is proposed in this paper. Talking heads synthesis can be considered as a learning problem of sequence-to-sequence mapping, which consists of audio as input and video as output. To synthesize talking heads, we use SAVEE database which consists of videos of multiple sentences speeches recorded from front of face. Audiovisual data can be considered as two parallel sequential data of audio and visual features and it is composed of continuous value. Thus, audio and visual features of our dataset are represented by a regression model. In this research, the regression model is trained with long short-term memory (LSTM) by minimizing mean squared error (MSE). Then, audio features are used as input and visual features are used as target of LSTM. Thereby, talking heads are synthesized from speech. Our method is proposed to use lower level audio features than phonemes and it enables to synthesize talking heads with expressions while existing researches which use phonemes as audio features only can synthesize neutral expression talking heads.With SAVEE database, we achieved the minimum MSE 17.03 on our testing dataset. In experiment, we use mel-frequency cepstral coefficient (MFCC), ΔMFCC and Δ 2 MFCC with energy as audio feature and active appearance model (AAM) on entire face region as visual feature.

DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering

“…Generating photo-realistic video portraits in line with any input audio stream has long been a popular research topic in com-puter graphics and vision [16,17,6]. Some methods aim at finding out the exact correspondence between audio and frames [39,11,56,27,59,15,34,38,40].…”

Neural Face Models for Example-Based Visual Speech Synthesis

“…Visual speech synthesis refers to the process of performing mouth animation according to a speech signal. This can be accomplished, for example, by performing a direct mapping from audio to animation parameters using a regression function [Cudeiro et al 2019;Hou et al 2016;Hussen Abdelaziz et al 2019;Karras et al 2017;Shimba et al 2015;Suwajanakorn et al 2017;Thies et al 2020;Zhou et al 2018]. For example [Shimba et al 2015] formulate the task as a sequence-to-sequence mapping with audio as input and video as output.…”

“…A similar approach is proposed by [Zhou et al 2018]. While they rely on similar audio features as [Shimba et al 2015], they use an animator centred face rig to represent facial expressions. This allows creating sequences of visual speech that can be edited and fine-tuned by a human animator.…”

Speaker-Independent Speech-Driven Visual Speech Synthesis using Domain-Adapted Acoustic Models

“…via phonemic transcription. For direct approaches, the conversion function typically involves some form of regression [16,24,26,27] or indexing a codebook of visual features using the corresponding features extracted from the acoustic speech [3,13]. For indirect approaches, the mapping function involves concatenation or interpolation of pre-existing data [5,7,9,21,29] or using a generative model [2,10,17].…”