The Effect of Real-Time Constraints on Automatic Speech Animation

Abstract

Machine learning has previously been applied successfully to speech-driven facial animation. To account for carry-over and anticipatory coarticulation a common approach is to predict the facial pose using a symmetric window of acoustic speech that includes both past and future context. Using future context limits this approach for animating the faces of characters in real-time and networked applications, such as online gaming. An acceptable latency for conversational speech is 200ms and typically network transmission times will consume a significant part of this. Consequently, we consider asymmetric windows by investigating the extent to which decreasing the future context effects the quality of predicted animation using both deep neural networks (DNNs) and bi-directional LSTM recurrent neural networks (BiLSTMs). Specifically we investigate future contexts from 170ms (fully-symmetric) to 0ms (fullyasymmetric). We find that a BiLSTM trained using 70ms of future context is able to predict facial motion of equivalent quality as a DNN trained with 170ms, while introducing increased processing time of only 5ms. Subjective tests using the BiLSTM show that reducing the future context from 170ms to 50ms does not significantly decrease perceived realism. Below 50ms, the perceived realism begins to deteriorate, generating a trade-off between realism and latency.

Speaker-Independent Speech Animation Using Perceptual Loss Functions and Synthetic Data

“…The system that generates visual features directly from an audio waveform follows the deep learning approach in [22], and maps a sequence of input MFCC vectors to an output sequence of AAM vectors. The approach in [22] was developed for a single speaker and uses an audio-visual speech database (KB-2k) for training that provides both acoustic and visual features. Our approach extends this system to become speaker independent.…”

“…In terms of context window sizes, a sequence of K A =33 MFCC features (340ms of audio) and K V B =3 visual features (100ms of video) gave best performance. Further details of the audio-to-visual speech model architecture can be found in [22].…”

“…This uses hand corrected phoneme annotations and so represents an ideal system. Second is a speaker-dependent system based on [22] that maps directly from acoustic features to AAM features using the ground-truth data (Audio-to-AAM). This is trained on the single speaker dataset and, although not appropriate for later speaker-independent operation, it serves as a useful baseline to evaluate the error introduced by mapping directly from audio compared with from phonemes.…”

“…However, large vocabulary, robust speech recognisers require broad contextual windows that span multiple words in order to exploit language models which improve decoding accuracy [20]. This introduces a significant time lag that makes real-time speech animation prohibitive, where tolerable delays are of the order 200ms [21], [22]. Conversely, audio-driven speech animation has been shown to generate realistic lip synchronisation in real-time, but in a speaker-dependent setting [2], [22].…”

“…This introduces a significant time lag that makes real-time speech animation prohibitive, where tolerable delays are of the order 200ms [21], [22]. Conversely, audio-driven speech animation has been shown to generate realistic lip synchronisation in real-time, but in a speaker-dependent setting [2], [22]. Our work leverages the advantages of both audio and text driven methods to achieve speaker-independent, real-time speech animation.…”