DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation

Ganin, Yaroslav; Kononenko, Daniil; Sungatullina, Diana; Lempitsky, Victor

doi:10.1007/978-3-319-46475-6_20

Cited by 120 publications

(87 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evaluation of Eye Reenactment. We compare our eye gaze reenactment strategy to the deep learning-based DeepWarp [Ganin et al 2016] approach, which only allows for gaze editing. As Fig.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Headon

et al. 2018

View full text Add to dashboard Cite

Fig. 1. Our novel HeadOn approach enables real-time reenactment of upper body motion, head pose, face expression, and eye gaze in human portrait videos. For synthesis of new photo-realistic video content, we employ a novel video-based rendering approach that builds on top of a fully controllable 3D actor model. The person-specific model is constructed from a short RGB-D calibration sequence and is driven by a real-time torso and face tracker.We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze. Given a short RGB-D video of the target actor, we automatically construct a personalized geometry proxy that embeds a parametric head, eye, and kinematic torso model. A novel realtime reenactment algorithm employs this proxy to photo-realistically map the captured motion from the source actor to the target actor. On top of the coarse geometric proxy, we propose a video-based rendering technique that composites the modified target portrait video via view-and pose-dependent texturing, and creates photo-realistic imagery of the target actor under novel torso and head poses, facial expressions, and gaze directions. To this end, we propose a robust tracking of the face and torso of the source actor. We extensively evaluate our approach and show significant improvements in enabling much greater flexibility in creating realistic reenacted output videos.

show abstract

“…Evaluation of Eye Reenactment. We compare our eye gaze reenactment strategy to the deep learning-based DeepWarp [Ganin et al 2016] approach, which only allows for gaze editing. As Fig.…”

Section: Resultsmentioning

confidence: 99%

“…11. Gaze redirection comparison: we compare our eye reenactment strategy (left) to the DeepWarp [Ganin et al 2016] gaze redirection approach (right). Note that DeepWarp merely modifies gaze direction, but does not perform a full reenactment of portrait videos.…”

Section: Resultsmentioning

confidence: 99%

Headon

et al. 2018

View full text Add to dashboard Cite

show abstract

“…More realistic redirected samples. Ganin et al [6] used a lighness correction refinement module on the gaze image redirected from the inverse warping field to produce a more realistic final redirected image. It indeed removed a lot of artifacts in our case.…”

Section: Discussionmentioning

confidence: 99%

“…proposed a pixel-wise replacement method using an eye flow tree and could synthesize realistic views with a gaze systematically redirected upwards by 10 to 15 degrees [13]. Then they updated the eye flow tree by a deep warping network trained on pairs of eye images corresponding to eye appearance before and after the redirection [6,12]. However, these methods require large amount of annotated data for training.…”

Section: Related Workmentioning

confidence: 99%

Improving Few-Shot User-Specific Gaze Adaptation via Gaze Redirection Synthesis

Liu

Odobez

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

As an indicator of human attention gaze is a subtle behavioral cue which can be exploited in many applications. However, inferring 3D gaze direction is challenging even for deep neural networks given the lack of large amount of data (groundtruthing gaze is expensive and existing datasets use different setups) and the inherent presence of gaze biases due to person-specific difference. In this work, we address the problem of person-specific gaze model adaptation from only a few reference training samples. The main and novel idea is to improve gaze adaptation by generating additional training samples through the synthesis of gaze-redirected eye images from existing reference samples. In doing so, our contributions are threefold: (i) we design our gaze redirection framework from synthetic data, allowing us to benefit from aligned training sample pairs to predict accurate inverse mapping fields; (ii) we proposed a self-supervised approach for domain adaptation; (iii) we exploit the gaze redirection to improve the performance of person-specific gaze estimation. Extensive experiments on two public datasets demonstrate the validity of our gaze retargeting and gaze estimation framework.

show abstract

“…smiling mouth) from one image to another [YWS*11]. Some other methods aim at manipulating eye gaze in 2D video [KPB*12, GKSL16] or editing 3D facial animation crafted by artists at the sequence level [LD08, MLD09]. The work of [YBS*12] achieves exaggeration, attenuation or replacement of facial expression in parts of 2D video.…”

Section: Related Workmentioning

confidence: 99%

Real‐Time Facial Expression Transformation for Monocular RGB Video

Deng

2018

Computer Graphics Forum

View full text Add to dashboard Cite

This paper describes a novel real‐time end‐to‐end system for facial expression transformation, without the need of any driving source. Its core idea is to directly generate desired and photo‐realistic facial expressions on top of input monocular RGB video. Specifically, an unpaired learning framework is developed to learn the mapping between any two facial expressions in the facial blendshape space. Then, it automatically transforms the source expression in an input video clip to a specified target expression through the combination of automated 3D face construction, the learned bi‐directional expression mapping and automated lip correction. It can be applied to new users without additional training. Its effectiveness is demonstrated through many experiments on faces from live and online video, with different identities, ages, speeches and expressions.

show abstract

DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation

Cited by 120 publications

References 20 publications

Headon

Headon

Improving Few-Shot User-Specific Gaze Adaptation via Gaze Redirection Synthesis

Real‐Time Facial Expression Transformation for Monocular RGB Video

Contact Info

Product

Resources

About