Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model

Deena, Salil; Hou, Shaobo; Galata, Aphrodite

doi:10.1145/1891903.1891942

Cited by 16 publications

(13 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A more lexible approach is to use a generative statistical model, such as GMMs [Luo et al 2014], switching linear dynamical systems [Englebienne et al 2007], switching shared Gaussian process dynamical models [Deena et al 2010], recurrent neural networks [Fan et al 2015], or hidden Markov models (HMMs) and their variants [Anderson et al 2013;Brand 1999;Fu et al 2005;Govokhina et al 2006;Schabus et al 2011;Wang et al 2012;Xie and Liu 2007]. During training of a HMM-based synthesiser, context-dependent decision trees cluster motion data and combine states with similar distributions to account for sparsity of the phonetic contexts in the training set.…”

Section: Related Workmentioning

confidence: 99%

A deep learning approach for generalized speech animation

et al. 2017

View full text Add to dashboard Cite

Fig. 1. A machine learning approach is used to learn a regression function mapping phoneme labels to speech animation. Our approach generates continuous, natural-looking speech animation for a reference face parameterization that can be retargeted to the face of any computer generated character.We introduce a simple and efective deep learning approach to automatically generate natural looking speech animation that synchronizes to input speech. Our approach uses a sliding window predictor that learns arbitrary nonlinear mappings from phoneme label input sequences to mouth movements in a way that accurately captures natural motion and visual coarticulation efects. Our deep learning approach enjoys several attractive properties: it runs in real-time, requires minimal parameter tuning, generalizes well to novel input speech sequences, is easily edited to create stylized and emotional speech, and is compatible with existing animation retargeting approaches. One important focus of our work is to develop an efective approach for speech animation that can be easily integrated into existing production pipelines. We provide a detailed description of our end-to-end approach, including machine learning design decisions. Generalized speech animation results are demonstrated over a wide range of animation clips on a variety of characters and voices, including singing and foreign language input. Our approach can also generate on-demand speech animation in real-time from user speech input.

show abstract

Section: Related Workmentioning

confidence: 99%

A deep learning approach for generalized speech animation

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Meanwhile, the assessment for the naturalness criterion with a scale of 1 to 5, where scale 1 means 'the motion of the mouth is very unrepresentative and the movement of the mouth is very rough', scale 2 means 'unrepresentative mouth motion and rough mouth movement changes', Scale 3 means 'relatively representative mouth movements and moderately gentle motion movements', scale 4 means 'representing mouth motion and subtle gesture motion', scale 5 means 'very representative mouth movements and very fine motion movements' change. The average respondent scores were calculated using Mean Opinion Score (MOS) formulated as equation (3).…”

Section: Resultsmentioning

confidence: 99%

“…Recently, natural and realistic facial animation is one of the most challenging research areas [3]. The effort to produce natural and realistic face animation is to add transition animations between pronunciations or apply dynamic viseme to the visual speech synthesis [4].…”

Section: Introductionmentioning

confidence: 99%

A Realistic Visual Speech Synthesis for Indonesian Using A Combination of Morphing Viseme and Syllable Concatenation Approach to Support Pronunciation Learning

Aripin¹,

Haryanto²

2018

Int. J. Emerg. Technol. Learn.

View full text Add to dashboard Cite

This study aims to build a realistic visual speech synthesis for Indonesian so that it can be used to learn Indonesian pronunciation. In this study, We used the combination of morphing viseme and syllable concatenation method. The morphing viseme method is a process of deformation from one viseme to another so that the animation of the mouth shape looks smoother. This method is used to create the transition of animation between viseme. The Syllable Concatenation method is used to assemble viseme based on certain syllable patterns. We built a syllable-based voice database as a basis for synchronization between syllables, speech and viseme models. The method proposed in this study consists of several stages, namely the formation of Indonesian viseme models, designing facial animation character, development of speech database, a synchronization process and subjective testing of the resulting application. Subjective tests were conducted on 30 respondents who assessed the suitability and natural movement of the mouth when uttering the Indonesian texts. The MOS (Mean Opinion Score) method is used to calculate the average of respondents' scores. The MOS calculation results for the criteria of Synchronization and naturalness are 4,283 and 4,107 on the scale of 1 to 5. This result shows that the level of Synchronization and naturalness of the synthesis of visual speech is more realistic. Therefore, the system can display the visualization of phoneme pronunciation to support learning Indonesian pronunciation.

show abstract

“…The closest approximation familiar to us is speech-driven facial animation [12]. We here discuss specific issues in using GPDMs as speech acoustic models, and propose an initialization scheme for sequential signals such as speech utterances.…”

Section: Implementing Gpdms For Speechmentioning

confidence: 99%

Gaussian process dynamical models for nonparametric speech representation and synthesis

Henter

Frean

Kleijn

2012

2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We propose Gaussian process dynamical models (GPDMs) as a new paradigm in acoustic models of speech. By using multidimensional, continuous state-spaces, the technique can overcome familiar limitations of discrete-state, HMM-based speech models. The added dimensions allow the state to represent and describe more than just temporal structure as systematic differences in mean, rather than as mere correlations in a residual (as HMMs with dynamics or AR-HMMs do). Owing to their basis in Gaussian processes, the models avoid restrictive parametric or linearity assumptions on signal structure. We outline GPDM theory, and describe model setup and initialization schemes relevant to speech applications. Experiments demonstrate subjectively better quality of synthesized speech than from comparable HMMs. In addition, there is evidence for unsupervised discovery of salient speech structure.

show abstract

Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model

Cited by 16 publications

References 25 publications

A deep learning approach for generalized speech animation

A deep learning approach for generalized speech animation

A Realistic Visual Speech Synthesis for Indonesian Using A Combination of Morphing Viseme and Syllable Concatenation Approach to Support Pronunciation Learning

Gaussian process dynamical models for nonparametric speech representation and synthesis

Contact Info

Product

Resources

About