The relightables

Abstract

built the hardware and infrastructure components. ‡ Authors contributed equally to this work. S. Fanello led the volumetric capture algorithm and pipeline implementation, G. Fyffe led the relightability features and storage infrastructure, C. Rhemann led the capture hardware and software development, J. Taylor led the engineering and algorithmic optimizations. § Equally last.

The Rocketbox Library and the Utility of Freely Available Rigged Avatars

“…Recent deep learning methods also using retrieved data can be used at many stages of the avatar creation process. Some methods have been successfully used to create avatars from pictures by recreating full 3D meshes from a photo (Hu et al, 2017;Saito et al, 2019), meshes from multiple cameras Collet et al (2015); Guo et al (2019), reduce the generated artifacts (Blanz and Vetter, 1999;Ichim et al, 2015), as well as to improve rigging (Weng et al, 2019). Deep learning methods can also generate completely new avatars that are not representations of existing people, by using adversarial networks (Karras et al, 2019).…”

“…Moreover, from a temporal point of view, we can also distinguish between capturing a single frame, and then performing rigging on the single frame, or capturing a sequence of these 3D representations, which is usually known as volumetric video. Finally, volumetric avatars can also be categorized as those that are captured offline (Collet et al, 2015;Guo et al, 2019) and played back as volumetric streams of data or those that are being captured and streamed in real-time (Orts-Escolano et al, 2016).…”

“…A traditional volumetric capture pipeline includes the following steps: image acquisition, image preprocessing, depth estimation, 3D reconstruction, texture UV atlas parameterization, and data compression. Some additional steps are usually performed to remove flickering effects in the final volumetric video, as mesh tracking (Guo et al, 2019) or non-rigid fusion (Dou et al, 2016(Dou et al, , 2017 in the case of offline approaches and real-time ones, respectively. More detailed information about state-of-the-art volumetric performance capture systems can be found in Guo et al (2019), Collet et al (2015), and Orts-Escolano et al (2016).…”

Investigating the Perceived Strengths and Limitations of Free-Viewpoint Video

“…Both of these techniques have issues with generalizability, however, as the selected mesh may struggle to represent varied or large shape changes (Casas et al, 2012;Collet et al, 2015). An alternative approach currently in development by Google combines FVV with lightstage techniques (Debevec et al, 2000), using timemultiplexed color gradient illumination to capture temporally consistent reflectance maps (Guo et al, 2019). While the results demonstrated using this technique are impressive, it represents an additional step up in complexity, and such techniques are not yet commercially available.…”

“…In this paper, we focus on the current generation of commercially available, high-quality FVV content. There are a number of lower-end commercially available FVV techniques, such as DepthKit (Depthkit, nd), as well as more advanced techniques in use in academic settings that are not currently available for commercial use (e.g., Guo et al, 2019). As it is important to reflect on the current technology landscape, we reference these techniques where appropriate.…”