Neural Rendering and Reenactment of Human Actor Videos

Liu, Lingjie; Xu, Weipeng; Zollhöfer, Michael; Kim, Hyeongwoo; Bernard, Florian; Habermann, Marc; Wang, Wenping; Theobalt, Christian

doi:10.1145/3333002

Cited by 159 publications

(124 citation statements)

References 67 publications

Supporting

Mentioning

124

Contrasting

Order By: Relevance

“…Recently, several performance cloning techniques proposed deep generative networks, trained to produce frames that contain the appearance of a target actor reenacting the motion of a driving actor [Aberman et al 2018;Chan et al 2018;Liu et al 2018].…”

Section: Performance Cloningmentioning

confidence: 99%

Learning character-agnostic motion for motion retargeting in 2D

Aberman

Lischinski

et al. 2019

ACM Trans. Graph.

View full text Add to dashboard Cite

Analyzing human motion is a challenging task with a wide variety of applications in computer vision and in graphics. One such application, of particular importance in computer animation, is the retargeting of motion from one performer to another. While humans move in three dimensions, the vast majority of human motions are captured using video, requiring 2D-to-3D pose and camera recovery, before existing retargeting approaches may be applied. In this paper, we present a new method for retargeting video-captured motion between different human performers, without the need to explicitly reconstruct 3D poses and/or camera parameters.In order to achieve our goal, we learn to extract, directly from a video, a high-level latent motion representation, which is invariant to the skeleton geometry and the camera view. Our key idea is to train a deep neural network to decompose temporal sequences of 2D poses into three components: motion, skeleton, and camera view-angle. Having extracted such a representation, we are able to re-combine motion with novel skeletons and camera views, and decode a retargeted temporal sequence, which we compare to a ground truth from a synthetic dataset.We demonstrate that our framework can be used to robustly extract human motion from videos, bypassing 3D reconstruction, and outperforming existing retargeting methods, when applied to videos in-the-wild. It also enables additional applications, such as performance cloning, video-driven cartoons, and motion retrieval.Webpage (code and data): https://motionretargeting2d.github.io/

show abstract

Section: Performance Cloningmentioning

confidence: 99%

Learning character-agnostic motion for motion retargeting in 2D

Aberman

Lischinski

et al. 2019

ACM Trans. Graph.

View full text Add to dashboard Cite

show abstract

“…Exact architectures and losses for such networks is an active area of research [14,31,33,65]. Very recent works [1,12,42,67] have used direct translation (with various modifications) to synthesize the view of a person for a fixed camera. We use the video-to-video variant of this approach [67] as a baseline for our method.…”

Section: Methodsmentioning

confidence: 99%

“…Over the last several months, several groups have presented results of neural modeling of full bodies [1,12,42,67]. While the presented results are very impressive, the approaches still require a large amount of training data.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Textured Neural Avatars

Shysheya

Ulyanov

Vakhitov

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

119

113

View full text Add to dashboard Cite

Figure 1: We propose a new model for neural rendering of humans. The model is trained for a single person and can produce renderings of this person from novel viewpoints (top) or in the new body pose (bottom) unseen during training. To improve generalization, our model retains explicit texture representation, which is learned alongside the rendering neural network. AbstractWe present a system for learning full-body neural avatars, i.e. deep networks that produce full-body renderings of a person for varying body pose and camera position. Our system takes the middle path between the classical graphics pipeline and the recent deep learning approaches that generate images of humans using image-to-image translation. In particular, our system estimates an explicit twodimensional texture map of the model surface. At the same time, it abstains from explicit shape modeling in 3D. Instead, at test time, the system uses a fully-convolutional network to directly map the configuration of body feature points w.r.t. the camera to the 2D texture coordinates of individual pixels in the image frame. We show that such a system is capable of learning to generate realistic renderings while being trained on videos annotated with 3D poses and foreground masks. We also demonstrate that maintaining an explicit texture representation helps our system to achieve better generalization compared to systems that use direct image-to-image translation.

show abstract

“…Video-tovideo translation has been used in many applications. For example, impressive results have been shown for the reenactment of the human head [Olszewski et al 2017], head and upper body , and the whole human body [Chan et al 2018;Liu et al 2018].…”

Section: Related Workmentioning

confidence: 99%