High fidelity facial animation capture and retargeting with contours

Bhat, K. Sham; Goldenthal, Rony; Ye, Yuting; Mallet, Ronald; Koperwas, Michael

doi:10.1145/2485895.2485915

Cited by 42 publications

(42 citation statements)

References 48 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With another purpose, but related to identifying facial expressions in synthetic faces, is the work of (Fernandes, Alves, Miranda, Queirós, & Orvalho, 2011) that presents a system to help children with autism to learn about facial expressions using CG characters. Some recent work focused in facial motion capture and real-time retargeting (Bhat, Goldenthal, Ye, Mallet, & Koperwas, 2013;H. Li, Yu, Ye, & Bregler, 2013) also discuss the importance of preserving microexpressions, although they do not guarantee it yet: the framerate of and inherent noise in the ordinary imaging devices used in this kind of application present technical challenges to robustly capturing microexpressions.…”

Section: Related Workmentioning

confidence: 99%

Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces

Queiroz

Musse

Badler

2014

Presence: Teleoperators and Virtual Environments

View full text Add to dashboard Cite

Due to varied personal, social, or even cultural situations, people sometimes conceal or mask their true emotions. These suppressed emotions can be expressed in a very subtle way by brief movements called microexpressions. We investigate human subjects' perception of hidden emotions in virtual faces, inspired by recent psychological experiments. We created animations with virtual faces showing some facial expressions and inserted brief secondary expressions in some sequences, in order to try to convey a subtle second emotion in the character. Our evaluation methodology consists of two sets of experiments, with three different sets of questions. The first experiment verifies that the accuracy and concordance of the participant's responses with synthetic faces matches the empirical results done with photos of real people in the paper by X.-b. Shen, Q. Wu, and X.-l. Fu, 2012, "Effects of the duration of expressions on the recognition of microexpressions," Journal of Zhejiang University Science B, 13(3), 221-230. The second experiment verifies whether participants could perceive and identify primary and secondary emotions in virtual faces. The third experiment tries to evaluate the participant's perception of realism, deceit, and valence of the emotions. Our results show that most of the participants recognized the foreground (macro) emotion and most of the time they perceived the presence of the second (micro) emotion in the animations, although they did not identify it correctly in some samples. This experiment exposes the benefits of conveying microexpressions in computer graphics characters, as they may visually enhance a character's emotional depth through subliminal microexpression cues, and consequently increase the perceived social complexity and believability (Shen, Wu, & Fu, 2012). The second experiment verifies if participants could perceive and identify primary and secondary emotions in virtual faces. The third tries to evaluate the participant's perception of realism, deceit and valence of the emotions. Our results show that most of the participants recognized the foreground (macro) emotion and most of the time they had perceived the presence of the second (micro) emotion in the animations, although they did not identify it correctly in some samples. This experiment exposes the benefits of conveying microexpressions in Computer Graphics characters, as they may visually enhance a character's emotional "depth" through subliminal microexpression cues and consequently increase its perceived social complexity and believability.

show abstract

Section: Related Workmentioning

confidence: 99%

Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces

Queiroz

Musse

Badler

2014

Presence: Teleoperators and Virtual Environments

View full text Add to dashboard Cite

show abstract

“…We believe that such a database can be used to determine a correct regression model without the need for calibration before each use. Similar to head mounted cameras used in production for motion capture [Bhat et al 2013], we attach an RGB-D camera to record the mouth region of the subject, but still intend to design a more ergonomic solution with either smaller and closer range cameras, or even alternative sensors. To advance the capture capabilities of our system, we hope to increase the number of strain gauges and combine our system with other sensors such as HMD integrated eye tracking systems [SMI 2014].…”

Section: Resultsmentioning

confidence: 99%

“…In the entertainment industry, facial performance capture is an established approach to improve the efficiency of animation production by reducing manual key-framing tasks and minimizing complex physical simulations of facial biomechanics [Terzopoulos and Waters 1990;Sifakis et al 2005]. To achieve the highest possible facial tracking fidelity, marker-based solutions, hand-assisted tracking, and multi-camera settings are still commonly employed [Bhat et al 2013;Bickel et al 2007;Pighin and Lewis 2006], while often requiring intensive computation.…”

Section: Previous Workmentioning

confidence: 99%

Facial performance sensing head-mounted display

et al. 2015

View full text Add to dashboard Cite

To enable immersive face-to-face communication in virtual worlds, the facial expressions of a user have to be captured while wearing a virtual reality head-mounted display. Because the face is largely occluded by typical wearable displays, we have designed an HMD that combines ultra-thin strain sensors with a head-mounted RGB-D camera for real-time facial performance capture and animation. AbstractThere are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing effective facial capture with traditional techniques. To advance virtual reality as a nextgeneration communication platform, we develop a novel HMD that enables 3D facial performance-driven animation in real-time. Our wearable system uses ultra-thin flexible electronic materials that are mounted on the foam liner of the headset to measure surface strain signals corresponding to upper face expressions. These strain signals are combined with a head-mounted RGB-D camera to enhance the tracking in the mouth region and to account for inaccurate HMD placement. To map the input signals to a 3D face model, we perform a single-instance offline training session for each person. For reusable and accurate online operation, we propose a short calibration step to readjust the Gaussian mixture distribution of the mapping before each use. The resulting animations are visually on par with cutting-edge depth sensor-driven facial performance capture systems and hence, are suitable for social interactions in virtual worlds.

show abstract

“…Kholgade et al [2011] developed a layered composition model of expressions for retargeting facial performances from the source model to a target character with dissimilar facial structure. More recently, blendshape mapping has been combined with facial tracking to animate a target character model [Bhat et al 2013;Bouaziz et al 2013;Li et al 2013]. Blendshape mapping, however, often requires skilled artists to manually create the facial rig to ensure the quality of output animation, but this is a slow, labor-intensive and costly process.…”

Section: Facial Expression Retargetingmentioning

confidence: 99%

“…In addition, the facial features around the eyes and mouth are especially important for high quality facial animation [Bhat et al 2013]. For example, commercial applications such as Live Driver [ImageMetrics ] have obtained impressive results for facial puppetry by tracking only those facial features around those regions.…”

Section: Problem Formulationmentioning

confidence: 99%

Controllable high-fidelity facial performance transfer

Chai²,

Liu

et al. 2014

ACM Trans. Graph.

View full text Add to dashboard Cite

Figure 1: Our system transfers high-fidelity facial animation data from a source facial model to a target facial model while allowing the user to control and edit the retargeted animation sequence in the spacetime domain. AbstractRecent technological advances in facial capture have made it possible to acquire high-fidelity 3D facial performance data with stunningly high spatial-temporal resolution. Current methods for facial expression transfer, however, are often limited to large-scale facial deformation. This paper introduces a novel facial expression transfer and editing technique for high-fidelity facial performance data. The key idea of our approach is to decompose high-fidelity facial performances into high-level facial feature lines, large-scale facial deformation and fine-scale motion details and transfer them appropriately to reconstruct the retargeted facial animation in an efficient optimization framework. The system also allows the user to quickly modify and control the retargeted facial sequences in the spatial-temporal domain. We demonstrate the power of our approach by transferring and editing high-fidelity facial animation data from high-resolution source models to a wide range of target models, including both human faces and non-human faces such as "monster" and "dog".

show abstract

High fidelity facial animation capture and retargeting with contours

Cited by 42 publications

References 48 publications

Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces

Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces

Facial performance sensing head-mounted display

Controllable high-fidelity facial performance transfer

Contact Info

Product

Resources

About