Playable universal capture

Borshukov, George; Montgomery, Jefferson; Werner, Witek; Ruff, Barry; Lau, James N.; Thuriot, Paul; Mooney, Patrick; Niekerk, Stefan Van; Raposo, Dave; Duprat, Jean-Luc; Hable, John; Kihlström, Håkan; Roizman, Daniel; Noone, Kevin J.; O’Connell, Jeff

doi:10.1145/1179849.1179884

Cited by 5 publications

(5 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To circumvent these limitations, several image‐based approaches like [8–17] have been developed. For example in [13], a low‐cost system for facial performance transfer is presented.…”

Section: Related Workmentioning

confidence: 99%

Interactive facial animation with deep neural networks

Paier

Hilsmann

Eisert

2020

IET Computer Vision

View full text Add to dashboard Cite

Creating realistic animations of human faces is still a challenging task in computer graphics. While computer graphics (CG) models capture much variability in a small parameter vector, they usually do not meet the necessary visual quality. This is due to the fact, that geometry‐based animation often does not allow fine‐grained deformations and fails in difficult areas (mouth, eyes) to produce realistic renderings. Image‐based animation techniques avoid these problems by using dynamic textures that capture details and small movements that are not explained by geometry. This comes at the cost of high‐memory requirements and limited flexibility in terms of animation because dynamic texture sequences need to be concatenated seamlessly, which is not always possible and prone to visual artefacts. In this study, the authors present a new hybrid animation framework that exploits recent advances in deep learning to provide an interactive animation engine that can be used via a simple and intuitive visualisation for facial expression editing. The authors describe an automatic pipeline to generate training sequences that consist of dynamic textures plus sequences of consistent three‐dimensional face models. Based on this data, they train a variational autoencoder to learn a low‐dimensional latent space of facial expressions that is used for interactive facial animation.

show abstract

“…To circumvent these limitations, several image‐based approaches like [8–17] have been developed. For example in [13], a low‐cost system for facial performance transfer is presented.…”

Section: Related Workmentioning

confidence: 99%

Interactive facial animation with deep neural networks

Paier

Hilsmann

Eisert

2020

IET Computer Vision

View full text Add to dashboard Cite

show abstract

“…oral cavity) or simply ignore them. Other works like [Borshukov et al 2006;Carranza et al 2003;Casas et al 2014;Dale et al 2011;Fechteler et al 2014Fechteler et al , 2016Kilner et al 2006;Lipski et al 2011;Paier et al 2015Paier et al , 2017, try to circumvent these limitations by employing image-based rendering approaches, where the facial performance is captured in geometry and texture space. [Dale et al 2011] present a low cost system for facial performance transfer.…”

Section: Related Work 21 Modelling Of Facial Expressionsmentioning

confidence: 99%

Neural Face Models for Example-Based Visual Speech Synthesis

Paier

Hilsmann

Eisert

2020

European Conference on Visual Media Production

View full text Add to dashboard Cite

“…oral cavity) or simply ignore them. Other works like [2,6,7,11,15,16,25,29,34,35], try to circumvent these limitations by employing image-based rendering approaches, where the facial performance is captured in geometry and texture space. In [11], Dale et al present a low cost system for facial performance transfer.…”

Section: Related Workmentioning

confidence: 99%

Neural Face Models for Example-Based Visual Speech Synthesis

Paier¹,

Hilsmann²,

Eisert³

2020

Preprint

View full text Add to dashboard Cite

Creating realistic animations of human faces with computer graphic models is still a challenging task. It is often solved either with tedious manual work or motion capture based techniques that require specialised and costly hardware. Example based animation approaches circumvent these problems by re-using captured data of real people. This data is split into short motion samples that can be looped or concatenated in order to create novel motion sequences. The obvious advantages of this approach are the simplicity of use and the high realism, since the data exhibits only real deformations. Rather than tuning weights of a complex face rig, the animation task is performed on a higher level by arranging typical motion samples in a way such that the desired facial performance is achieved. Two difficulties with example based approaches, however, are high memory requirements as well as the creation of artefact-free and realistic transitions between motion samples. We solve these problems by combining the realism and simplicity of example-based animations with the advantages of neural face models. Our neural face model is capable of synthesising high quality 3D face geometry and texture according to a compact latent parameter vector. This latent representation reduces memory requirements by a factor of 100 and helps creating seamless transitions between concatenated motion samples. In this paper, we present a marker-less approach for facial motion capture based on multi-view video. Based on the captured data, we learn a neural representation of facial expressions, which is used to seamlessly concatenate facial performances during the animation procedure. We demonstrate the effectiveness of our approach by synthesising mouthings for Swiss-German sign language based on viseme query sequences.

show abstract

Playable universal capture

Cited by 5 publications

References 2 publications

Interactive facial animation with deep neural networks

Interactive facial animation with deep neural networks

Neural Face Models for Example-Based Visual Speech Synthesis

Neural Face Models for Example-Based Visual Speech Synthesis

Contact Info

Product

Resources

About