Towards Learning a Realistic Rendering of Human Behavior

Esser, Patrick; Haux, Johannes; Milbich, Timo; Ommer, Bj orn

doi:10.1007/978-3-030-11012-3_32

Cited by 15 publications

(10 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While some approaches have shown convincing results for the facial area [Kim et al 2018a;Lombardi et al 2018], creating photo-real images of the entire human is still a challenge. Most of the methods, which target synthesizing entire humans, learn an image-to-image mapping from renderings of a skeleton [Chan et al 2019;Esser et al 2018;Pumarola et al 2018;Si et al 2018], depth map [Martin-Brualla et al 2018], dense mesh [Liu et al 2020b[Liu et al , 2019bWang et al 2018a] or joint position heatmaps [Aberman et al 2019], to real images. Among these approaches, the most related work [Liu et al 2020b] achieves better temporally-coherent dynamic textures by first learning fine scale details in texture space and then translating the rendered mesh with dynamic textures into realistic imagery.…”

Section: Related Workmentioning

confidence: 99%

Real-time deep dynamic characters

Habermann

Liu

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit spacetime coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.

show abstract

Section: Related Workmentioning

confidence: 99%

Real-time deep dynamic characters

Habermann

Liu

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

show abstract

“…Another line of work on animating photorealistic human rendering skips the complicated 3D geometry modeling, and rather focuses on synthesizing photorealistic images or videos by solving an image translation problem, e.g., learning the mapping function from joint heatmaps [Aberman et al 2019], rendered skeleton Esser et al 2018;Pumarola et al 2018;Shysheya et al 2019;Si et al 2018], or rendered meshes [Liu et al 2019c,b;Sarkar et al 2020;Thies et al 2019;, to real images. Although these methods often do generate plausible images, they tend to have challenges in generalizing to different poses, due to complex articulations of the human body and dynamics of the clothing deformations.…”

Section: Avatar Modelingmentioning

confidence: 99%

Driving-signal aware full-body avatars

Bagautdinov

Simon

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

We present a learning-based method for building driving-signal aware full-body avatars. Our model is a conditional variational autoencoder that can be animated with incomplete driving signals, such as human pose and facial keypoints, and produces a high-quality representation of human geometry and view-dependent appearance. The core intuition behind our method is that better drivability and generalization can be achieved by disentangling the driving signals and remaining generative factors, which are not available during animation. To this end, we explicitly account for information deficiency in the driving signal by introducing a latent space that exclusively captures the remaining information, thus enabling the imputation of the missing factors required during full-body animation, while remaining faithful to the driving signal. We also propose a learnable localized compression for the driving signal which promotes better generalization, and helps minimize the influence of global chance-correlations often found in real datasets. For a given driving signal, the resulting variational model produces a compact space of uncertainty for missing factors that allows for an imputation strategy best suited to a particular application. We demonstrate the efficacy of our approach on the challenging problem of full-body animation for virtual telepresence with driving signals acquired from minimal sensors placed in the environment and mounted on a VR-headset.

show abstract

“…Very recently, multiple approaches for video-based human performance cloning have been proposed [4], [5], [6], [8], [36], [37], [38] that output realistic video sequences. These approaches learn complex image-to-image mappings, i.e., from renderings of a skeleton [4], [36], [37], [38], dense mesh [6], [8], or joint position heatmaps [5], to real images. Liu et al [8] proposed to translate simple synthetic computer graphics renderings of a human character into realistic imagery.…”

Section: Related Workmentioning

confidence: 99%

Learning Dynamic Textures for Neural Rendering of Human Actors

Liu

Habermann

et al. 2021

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Synthesizing realistic videos of humans using neural networks has been a popular alternative to the conventional graphics-based rendering pipeline due to its high efficiency. Existing works typically formulate this as an image-to-image translation problem in 2D screen space, which leads to artifacts such as over-smoothing, missing body parts, and temporal instability of fine-scale detail, such as pose-dependent wrinkles in the clothing. In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. More specifically, our method relies on the combination of two convolutional neural networks (CNNs). Given the pose information, the first CNN predicts a dynamic texture map that contains time-coherent high-frequency details, and the second CNN conditions the generation of the final video on the temporally coherent output of the first CNN. We demonstrate several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.

show abstract

Towards Learning a Realistic Rendering of Human Behavior

Cited by 15 publications

References 35 publications

Real-time deep dynamic characters

Real-time deep dynamic characters

Driving-signal aware full-body avatars

Learning Dynamic Textures for Neural Rendering of Human Actors

Contact Info

Product

Resources

About