NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads

Abstract

We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps. To this end, we propose a new multi-view capture setup composed of 16 calibrated machine vision cameras that record time-synchronized images at 7.1 MP resolution and 73 frames per second. With our setup, we collect a new dataset of over 4700 high-resolution, high-framerate sequences of more than 220 human heads, from which we introduce a new human head reconstruction benchmark 1 . The recorded sequences cover a wide range of facial dynamics, including head motions, natural expressions, emotions, and spoken language. In order to reconstruct high-fidelity human heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles (NeRSemble). We represent scene dynamics by combining a deformation field and an ensemble of 3D multi-resolution hash encodings. The deformation field allows for precise modeling of simple scene movements, while the ensemble of hash encodings helps to represent complex dynamics. As a result, we obtain radiance field representations of human heads that capture motion over time and facilitate re-rendering of arbitrary novel viewpoints. In a series of experiments, we explore the design choices of our method and demonstrate that our approach outperforms state-of-the-art dynamic radiance field approaches by a significant margin.

State of the Art on Diffusion Models for Visual Computing

“…Last but not least, several datasets are used in audio‐driven face synthesis: Examples are Fanelli et al . [FGR*10], VOCAset [CBL*19], Voxceleb [NCXZ20] and NeRSemble [KQG*23b]. They provide video clips [FGR*10, NCXZ20, KQG*23b], dense and deformable 3D facial geometry [FGR*10, CBL*19] or deformable NeRFs [KQG*23b] and the corresponding synchronised audio [FGR*10,CBL*19,NCXZ20,KQG*23b].…”

“…[FGR*10], VOCAset [CBL*19], Voxceleb [NCXZ20] and NeRSemble [KQG*23b]. They provide video clips [FGR*10, NCXZ20, KQG*23b], dense and deformable 3D facial geometry [FGR*10, CBL*19] or deformable NeRFs [KQG*23b] and the corresponding synchronised audio [FGR*10,CBL*19,NCXZ20,KQG*23b]. The number of sequences in these datasets varies from hundreds [CBL*19] (12 subjects) and thousands (14 [FGR*10] and 220 [KQG*23b] subjects) to over 10 6 [NCXZ20] (7 k subjects).…”

“…As literature exploring the capabilities of this dataset is still in its infancy, it is too soon to predict the impact it will have on the field. At the same time, higher‐fidelity animation sequences from 3D scans or multi‐view capture setups have been released in the context of human faces [GKG*23, KQG*23a] and bodies [PZX*21,IRG*23,LHR*21,HLX*21,HXZ*20,XCZ*18]. Although these datasets are of high quality for each captured instance, the costly recording process limits the largest of such datasets to several hundred captured people.…”