Recent Advances in Facial Appearance Capture

Abstract

Facial appearance capture is now firmly established within academic research and used extensively across various application domains, perhaps most prominently in the entertainment industry through the design of virtual characters in video games and films. While significant progress has occurred over the last two decades, no single survey currently exists that discusses the similarities, differences, and practical considerations of the available appearance capture techniques as applied to human faces. A central difficulty of facial appearance capture is the way light interacts with skin—which has a complex multi‐layered structure—and the interactions that occur below the skin surface can, by definition, only be observed indirectly. In this report, we distinguish between two broad strategies for dealing with this complexity. “Image‐based methods” try to exhaustively capture the exact face appearance under different lighting and viewing conditions, and then render the face through weighted image combinations. “Parametric methods” instead fit the captured reflectance data to some parametric appearance model used during rendering, allowing for a more lightweight and flexible representation but at the cost of potentially increased rendering complexity or inexact reproduction. The goal of this report is to provide an overview that can guide practitioners and researchers in assessing the tradeoffs between current approaches and identifying directions for future advances in facial appearance capture.

State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications

“…Alongside geometry reconstruction, appearance capture is also of great importance. A recent report by Klehm et al [KRP*15] outlines related methods for capturing the appearance of faces, with the goal of re‐rendering in CG. Facial capture is closely related to the problem of non‐rigid surface registration, where the goal is to align a specific surface to images or 3D scans, a topic that is covered in detail in recent tutorials [BTP14, BTLP16].…”

“…In addition, faces also exhibit a certain amount of transmission and subsurface scattering. An in‐depth discussion of skin appearance modeling is beyond the scope of this state‐of‐the‐art report and we would like to refer to [KRP*15] for more details. In the context of monoculuar face capture, most work relies on simple appearance models that approximate real face appearance.…”

Appearance Modelling of Living Human Tissues

“…[BW17, Che15, IGAJG15]), just to name a few. In fact, a comprehensive review of the extensive body of work aiming at the rendering of skin would merit a state‐of‐the‐art report in its own right, contributing to recent efforts as in [KRP*15]. Clearly, such a review would be beyond the scope of this paper centred on how the distinct representations of human tissues affect the modelling of their appearance.…”

Interactive Realistic Digital Avatars - Revisiting the Uncanny Valley

“…Achieving a realistic human face in a non-interactive environment is only just being addressed at the high end of feature film effects [17]. Excellent work has been done in producing high quality facial rendering and animation [37], but most cannot be rendered in real-time.…”

“…While the realization of such agents involves various aspects such as responsiveness, appropriateness to task, accuracy in user intent, and quality of computer responses, one key aspect is the visual presentation of the agent itself. In this paper we study the effect of interactive digital avatars with photo-realistic human faces, based on technology and techniques that are only just becoming available and that originate from the field of feature film effects [17].…”