PINA: Learning a Personalized Implicit Neural Avatar from a Single RGB-D Video Sequence

Abstract

Figure 1. We propose PINA, a method to acquire personalized and animatable neural avatars from RGB-D videos. Left: our method uses only a single sequence, captured via a commodity depth sensor. The depth frames are noisy and contain only partial views of the body. Middle: Using global optimization, we fuse these partial observations into an implicit surface representation that captures geometric details, such as loose clothing. The shape is learned alongside a pose-independent skinning field, supervised only via depth observations. Right: The learned avatar can be animated with realistic articulation-driven surface deformations and generalizes to novel unseen poses.

Mesh Strikes Back: Fast and Efficient Human Reconstruction from RGB videos

“…These approaches use SMPL as a prior to unpose the human body across multiple frames by transforming the rays from observation space to canonical space which is then rendered using a NeRF. PINA [18] learns a SDF and a learned deformation field to create an animatable avatar from an RGB-D image sequence. Chen et al [15] used a polygon rasterization pipeline to speed up NeRF rendering as a post-process; however, their approach does not reduce NeRF training times.…”

“…Rendering the visual hull of both objects gives us the same set of binary silhouettes for all camera angles, making it ambiguous for any optimization scheme to recover a unique mesh from the set of silhouettes. This ambiguity necessitates the use of auxiliary inputs, e.g., depth or normals [18,69,47]. Now consider the same scenario, but with the same overlaid texture on both objects.…”

“…3.1). To overcome this ambiguity, recent methods augment silhouettes with depths or normals [69,18]. However, obtaining ground truth depths or normals is expensive, and prediction can be error-prone.…”

gDNA: Towards Generative Detailed Neural Avatars

“…To overcome the topology and resolution limitations of meshes, other representations, including point clouds [35,37,65], implicit surfaces [12,47,52,55,58,60], and radiance fields [32,45,49,56,63], have been explored. In particular, neural implicit surface representations have emerged as a powerful tool to model 3D (clothed) human shapes [6,15,17,20,21,30,41,50,51,62,66,67] due to their topological flexibility and resolution independence. Recent work [12,52,58] uses implicit surfaces to learn human avatars for a single subject, wearing a specific garment.…”

X-Avatar: Expressive Human Avatars

“…This approach has been shown to generalize to arbitrary poses. While SNARF [12] only models the major body bones, other works have focused on creating implicit models of the face [21,64], the hands [16], or how to model humans that appear in garments [19] and how to additionally capture appearance [47,52]. Although neural implicit avatars hold great promise, to date no model exists that holistically captures the body and all the parts that are important for human expressiveness jointly.…”