Practical SVBRDF acquisition of 3D objects with unstructured flash photography

Abstract

Capturing spatially-varying bidirectional reflectance distribution functions (SVBRDFs) of 3D objects with just a single, hand-held camera (such as an off-the-shelf smartphone or a DSLR camera) is a difficult, open problem. Previous works are either limited to planar geometry, or rely on previously scanned 3D geometry, thus limiting their practicality. There are several technical challenges that need to be overcome: First, the built-in flash of a camera is almost colocated with the lens, and at a fixed position; this severely hampers sampling procedures in the light-view space. Moreover, the near-field flash lights the object partially and unevenly. In terms of geometry, existing multiview stereo techniques assume diffuse reflectance only, which leads to overly smoothed 3D reconstructions, as we show in this paper. We present a simple yet powerful framework that removes the need for expensive, dedicated hardware, enabling practical acquisition of SVBRDF information from real-world, 3D objects with a single, off-the-shelf camera with a built-in flash. In addition, by removing the diffuse reflection assumption and leveraging instead such SVBRDF information, our method outputs high-quality 3D geometry reconstructions, including more accurate high-frequency details than state-of-the-art multiview stereo techniques. We formulate the joint reconstruction of SVBRDFs, shading normals, and 3D geometry as a multi-stage, iterative inverse-rendering reconstruction pipeline. Our method is also directly applicable to any existing multiview 3D reconstruction technique. We present results of captured objects with complex geometry and reflectance; we also validate our method numerically against other existing approaches that rely on dedicated hardware, additional sources of information, or both.

Progressive Acquisition of SVBRDF and Shape in Motion

“…However, the building cost of such systems is too high to make the acquisition process not available for casual users to have access to this acquisition process. To resolve this issue, practical methods using a single camera have been introduced [AWL15, RPG16, HSL*17, RRFG17, WZ15, WWZ16, SWK19, PNS18, NLGK18]. These methods can capture material appearance by inferring diffuse and specular appearance parameters from multiple observations with different view/light angles.…”

“…The state‐of‐the‐art methods for estimating material appearance have focused on SVBRDF of static objects [PNS18, NLGK18] or only diffuse albedo of dynamic objects [GXY*17]. As we estimate per‐vertex motion and shape, we then estimate complete SVBRDF parameters per vertex in a progressive way through our novel optimization method.…”

“…In contrast, we were motivated to devise a practical acquisition solution without requiring any specialized hardware setup, such as a mechanical gantry with two robotic arms or a multiview camera‐light stage. To this end, we decided to make use of a conventional RGBD camera for our acquisition setup, following the trend of state‐of‐the‐art practical techniques [AWL15, RPG16, HSL*17, RRFG17, WZ15, WWZ16, PNS18, NLGK18].…”

“…In addition, the traditional material acquisition methods [GHP*08, HLZ10, NLW*16, TFG*13, SSWK13, FHW*11, TAL*07, LWS*13] require several hours to capture input images of rigid objects. Our novel inverse rendering framework allows us to estimate SVBRDF parameters and shape information progressively in interactive time as we build our framework by combining the recent advances of fusion‐based methods [NFS15, IZN*16, GXY*17] and the practical inverse rendering technique that captures SVBRDF with active illumination [NLGK18, WZ15]. Our progressive acquisition approach does not need to wait for several hours to capture input images.…”

Deep reflectance fields

“…The approach of Gotardo et al [2018] for dynamic appearance estimation extracts SVBRDF (diffuse and specular) and geometry (also fine scale) from images captured under uniform lighting, but their approach is restricted to the skin region. Recently, multiple works have also focused on the challenging problem of extracting the SVBRDF from a single image using a flash [Li et al 2018a,b;Nam et al 2018]. Since, all model-based approaches use hand-crafted priors, they are typically limited to specific parts of the human body and only handle these in isolation.…”