HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars

Abstract

We propose a novel neural rendering pipeline, Hybrid Volumetric-Textural Rendering (HVTR), which synthesizes virtual human avatars from arbitrary poses efficiently and at high quality. First, we learn to encode articulated human motions on a dense UV manifold of the human body surface. To handle complicated motions (e.g., self-occlusions), we then leverage the encoded information on the UV manifold to construct a 3D volumetric representation based on a dynamic pose-conditioned neural radiance field. While this allows us to represent 3D geometry with changing topology, volumetric rendering is computationally heavy. Hence we employ only a rough volumetric representation using a pose-conditioned downsampled neural radiance field (PD-NeRF), which we can render efficiently at low resolutions. In addition, we learn 2D textural features that are fused with rendered volumetric features in image space. The key advantage of our approach is that we can then convert the fused features into a high resolution, high quality avatar by a fast GAN-based textural renderer. We demonstrate that hybrid rendering enables HVTR to handle complicated motions, render high quality avatars under usercontrolled poses/shapes and even loose clothing, and most importantly, be fast at inference time. Our experimental results also demonstrate state-of-the-art quantitative results. More results are available at our project page: https: // www. cs. umd. edu/ ~taohu/ hvtr/ .

Learning Neural Volumetric Representations of Dynamic Humans in Minutes

“…Recently, some methods [9,46,69,86] model the shapes of dynamic humans as implicit neural representations and attempt to optimize them from human scans. Another line of works [28,30,34,36,40,56,58,65,91,92,94,[100][101][102]104] exploits dynamic implicit neural representations and differentiable renderers to reconstruct 3D human models from videos. To represent dynamic humans, Neural Actor [40] augments the neural radiance field with the linear blend skinning model [35].…”

Human Performance Modeling and Rendering via Neural Animated Mesh

“…More recently, [Li et al 2020;Park et al 2020;Pumarola et al 2021;] extend neural radiance field [Mildenhall et al 2020] into the dynamic setting. [Hu et al 2021;Peng et al 2021a,b;] utilize the human prior SMPL [Loper et al 2015] model as an anchor and use linear blend skinning algorithm to warp the radiance field. Furthermore, [Jiang et al 2022a;Sun et al 2021] extend the dynamic neural rendering and blending into the humanobjection interaction scenarios.…”