LookinGood

Abstract

Fig. 1. LookinGood leverages recent advances in real-time 3D performance capture and machine learning to re-render high quality novel viewpoints of a captured scene. A textured 3D reconstruction is first rendered to a novel viewpoint. Due to imperfections in geometry and low-resolution texture, the 2D rendered image contains artifacts and is low quality. Therefore we propose a deep learning technique that takes these images as input and generates more visually enhanced re-rendering. The system is specifically designed for VR and AR headsets, and accounts for consistency between two stereo views.Motivated by augmented and virtual reality applications such as telepresence, there has been a recent focus in real-time performance capture of humans under motion. However, given the real-time constraint, these systems often suffer from artifacts in geometry and texture such as holes and noise in the final rendering, poor lighting, and low-resolution textures. We take the novel approach to augment such real-time performance capture systems with a deep architecture that takes a rendering from an arbitrary viewpoint, and jointly performs completion, super resolution, and denoising of the imagery in real-time. We call this approach neural (re-)rendering, and * Authors equally contributed to this work. our live system "LookinGood". Our deep architecture is trained to produce high resolution and high quality images from a coarse rendering in real-time. First, we propose a self-supervised training method that does not require manual ground-truth annotation. We contribute a specialized reconstruction error that uses semantic information to focus on relevant parts of the subject, e.g. the face. We also introduce a salient reweighing scheme of the loss function that is able to discard outliers. We specifically design the system for virtual and augmented reality headsets where the consistency between the left and right eye plays a crucial role in the final user experience. Finally, we generate temporally stable results by explicitly minimizing the difference between two consecutive frames. We tested the proposed system in two different scenarios: one involving a single RGB-D sensor, and upper body reconstruction of an actor, the second consisting of full body 360 • capture. Through extensive experimentation, we demonstrate how our system generalizes across unseen sequences and subjects. The supplementary video is available at http://youtu.be/Md3tdAKoLGU.

Real-time deep dynamic characters

“…While some approaches have shown convincing results for the facial area [Kim et al 2018a;Lombardi et al 2018], creating photo-real images of the entire human is still a challenge. Most of the methods, which target synthesizing entire humans, learn an image-to-image mapping from renderings of a skeleton [Chan et al 2019;Esser et al 2018;Pumarola et al 2018;Si et al 2018], depth map [Martin-Brualla et al 2018], dense mesh [Liu et al 2020b[Liu et al , 2019bWang et al 2018a] or joint position heatmaps [Aberman et al 2019], to real images. Among these approaches, the most related work [Liu et al 2020b] achieves better temporally-coherent dynamic textures by first learning fine scale details in texture space and then translating the rendered mesh with dynamic textures into realistic imagery.…”

“…Different from differentiable rendering, neural rendering makes almost no assumptions about the physical model and uses neural networks to learn the rendering process from data to synthesize photo-realistic images. Some neural rendering methods [Aberman et al 2019;Chan et al 2019;Kim et al 2018b;Liu et al 2020bLiu et al , 2019bMa et al 2017Ma et al , 2018Martin-Brualla et al 2018;Pumarola et al 2018;Shysheya et al 2019;Siarohin et al 2018;Thies et al 2019;Yoon et al 2020] employ image-to-image translation networks [Isola et al 2017;Wang et al 2018a,b] to augment the quality of the rendering. However, most of these methods suffer from view and/or temporal inconsistency.…”

Rig-space Neural Rendering

“…To circumvent both the limitation to low resolution images associated with adversarial learning, and to remove ambiguities due to flat 2d maps (e.g. 2d skeletons missing 3d surface orientation), methods have utilized paired images produced by first rendering coarse geometry [18,14,19,15,20] and then translating to the corresponding real-world counter-part.…”

“…The rendered geometry allows to condition the network with less ambiguous data: a UV map [19], a low resolution avatar with patterned clothing [14], a rendered 3d skeleton [15], or noisy captured albedo and normals [18,20]. All these methods first render the 3d geometry and thus require creating and manipulating this intermediate object in their pipeline.…”

“…Training with these losses leads to images that can remain blurry. To increase details, we experimented with losses from related work such a saliency loss [18] and multi-scale losses such as the perceptual loss [23]. We obtained marginal increase in detail with these experiments.…”