Skin microstructure deformation with displacement map convolution

Nagano, Koki; Fyffe, Graham; Alexander, Oleg; Barbič, Jernej; Li, Hao; Ghosh, Abhijeet; Debevec, Paul

doi:10.1145/2766894

Cited by 40 publications

(21 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…instead applied complementary spherical gradient illumination based alignment of Wilson et al [2010] in conjunction with high speed photography to acquire longer facial performance sequences, and employed a heuristics based diffuse-specular separation on the acquired data to obtain albedo and normal maps for rendering. Nagano et al [2015] have extended [Graham et al 2013] to acquire microgeometry of various skin patches under stretch and compression and employed the acquired data for efficient real-time rendering of dynamic facial microgeometry using texture space filtering. For true video-rate dynamic capture, have proposed employing spectral multiplexing with polarized spherical gradient illumination (using an RGB LED sphere) for facial performance capture.…”

Section: Related Workmentioning

confidence: 99%

Single-shot high-quality facial geometry and skin appearance capture

et al. 2020

Self Cite

View full text Add to dashboard Cite

We propose a new light-weight face capture system capable of reconstructing both high-quality geometry and detailed appearance maps from a single exposure. Unlike currently employed appearance acquisition systems, the proposed technology does not require active illumination and hence can readily be integrated with passive photogrammetry solutions. These solutions are in widespread use for 3D scanning humans as they can be assembled from off-the-shelf hardware components, but lack the capability of estimating appearance. This paper proposes a solution to overcome this limitation, by adding appearance capture to photogrammetry systems. The only additional hardware requirement to these solutions is that a subset of the cameras are cross-polarized with respect to the illumination, and the remaining cameras are parallel-polarized. The proposed algorithm leverages the images with the two different polarization states to reconstruct the geometry and to recover appearance properties. We do so by means of an inverse rendering framework, which solves per texel diffuse albedo, specular intensity, and high-resolution normals, as well as global specular roughness considering the subsurface scattering nature of skin. We show results for a variety of human subjects of different ages and skin typology, illustrating how the captured fine-detail skin surface and subsurface scattering effects lead to realistic renderings of their digital doubles, also in different illumination conditions.

show abstract

Section: Related Workmentioning

confidence: 99%

Single-shot high-quality facial geometry and skin appearance capture

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Another line of previous research focuses on capturing microgeometry explicitly using visible light [Graham et al 2013;Nagano et al 2015;Nam et al 2016]. Unlike this prior work, we do not try to explicitly reconstruct the microgeometry of the surface, but instead measure a slice of the local BRDF which is then used to infer the GNDF.…”

Section: Related Workmentioning

confidence: 99%

Learning generative models for rendering specular microgeometry

Кuznetsov

Hašan

et al. 2019

ACM Trans. Graph.

View full text Add to dashboard Cite

from synthetic or measured examples, by training a generative adversarial network (GAN). A key challenge in applying GAN synthesis to spatially varying BRDFs is evaluating the reflectance for a single location and direction without the cost of evaluating the whole hemisphere. We resolve this using a novel method for partial evaluation of the generator network. We are also able to control large-scale spatial texture using a conditional GAN approach. The benefits of our approach include the ability to synthesize spatially large results without repetition, support for learning from measured data, and evaluation performance independent of the complexity of the dataset synthesis or measurement. CCS Concepts: • Computing methodologies → Rendering.

show abstract

“…Gotardo et al [2015] propose a simpler binocular setup with spectral and temporal multiplexing of nine light sources to compute dynamic albedo and normal maps, but only diffuse reflectance is modeled. Finally, Nagano et al [2015] have acquired microgeometry of various skin patches under stretch and compression (using polarized spherical gradient illumination) and employed the acquired data for building an efficient real-time rendering technique for dynamic facial microgeometry using texture space filtering of the neutral displacement map. Our work is related to this but we estimate skin surface geometry changes due to stretching and compression at the scale of mesostructure.…”

Section: Practical Dynamic Facial Appearance Modeling and Acquisitionmentioning

confidence: 99%

Practical dynamic facial appearance modeling and acquisition

et al. 2018

Self Cite

View full text Add to dashboard Cite

We present a method to acquire dynamic properties of facial skin appearance, including dynamic diffuse albedo encoding blood flow, dynamic specular intensity, and per-frame high resolution normal maps for a facial performance sequence. The method reconstructs these maps from a purely passive multi-camera setup, without the need for polarization or requiring temporally multiplexed illumination. Hence, it is very well suited for integration with existing passive systems for facial performance capture. To solve this seemingly underconstrained problem, we demonstrate that albedo dynamics during a facial performance can be modeled as a combination of: (1) a static, high-resolution base albedo map, modeling full skin pigmentation; and (2) a dynamic, one-dimensional component in the CIE L*a*b* color space, which explains changes in hemoglobin concentration due to blood flow. We leverage this albedo subspace and additional constraints on appearance and surface geometry to also estimate specular reflection parameters and resolve high-resolution normal maps with unprecedented detail in a passive capture system. These constraints are built into an inverse rendering framework that minimizes the difference of the rendered face to the captured images, incorporating constraints from multiple views for every texel on the face. The presented method is the first system capable of capturing high-quality dynamic appearance maps at full resolution and video framerates, providing a major step forward in the area of facial appearance acquisition.

show abstract

Skin microstructure deformation with displacement map convolution

Cited by 40 publications

References 44 publications

Single-shot high-quality facial geometry and skin appearance capture

Single-shot high-quality facial geometry and skin appearance capture

Learning generative models for rendering specular microgeometry

Practical dynamic facial appearance modeling and acquisition

Contact Info

Product

Resources

About