Interactive Visual Hull Refinement for Specular and Transparent Object Surface Reconstruction

Abstract

In this paper we present a method of using standard multi-view images for 3D surface reconstruction of nonLambertian objects. We extend the original visual hull concept to incorporate 3D cues presented by internal occluding contours, i.e., occluding contours that are inside the object's silhouettes. We discovered that these internal contours, which are results of convex parts on an object's surface, can lead to a tighter fit than the original visual hull. We formulated a new visual hull refinement scheme -Locally Convex Carving that can completely reconstruct concavity caused by two or more intersecting convex surfaces.In addition we develop a novel approach for contour tracking given labeled contours in sparse key frames. It is designed specifically for highly specular or transparent objects, for which assumptions made in traditional contour detection/tracking methods, such as highest gradient and stationary texture edges, are no longer valid. It is formulated as an energy minimization function where several novel terms are developed to increase robustness.Based on the two core algorithms, we have developed an interactive system for 3D modeling. We have validated our system, both quantitatively and qualitatively, with four datasets of different object materials. Results show that we are able to generate visually pleasing models for very challenging cases.

Full 3D reconstruction of transparent objects

“…To estimate the geometry of transparent objects, the shape-from-distortion techniques [Ben-Ezra and Nayar 2003;Tanaka et al 2016;Wetzstein et al 2011] focus on analyzing known or unknown distorted background patterns. Zuo et al [2015] incorporate internal occluding contours into traditional shape-fromsilhouette methods, and propose a visual hull refinement scheme. It is also possible to reconstruct transparent objects by capturing exterior specular highlights [Morris and Kutulakos 2007;Yeung et al 2011] known as shape-from-reflectance.…”

Optimal Billboard Deformation via 3D Voxel for Free-Viewpoint System

“…There is a trend for, some conventional methods to focus on refining the shape of obtained visual hulls. For example, one of the latest methods [22] can represent the shape of occluding contours that are inside the object's silhouettes based on the assumption that the target being recorded is non-Lambertian object. Another of the latest methods [9] has proposed a new 3D primitive, Convex Bricks, to accurately express the object's shape and reduce the number of polygons.…”

Dense Reconstruction of Transparent Objects by Altering Incident Light Paths Through Refraction

“…Following the same fashion, Tsai et al [33] demonstrated two depth-normal ambiguities for transparent object recovery assuming the light path refracts exactly twice. In [38], Zuo et al developed an interactive specular and transparent object reconstruction system based on visual hull refinement given the silhouettes under multiple views and labeled contours of the object in sparse key frames. In [28], Qian et al introduced a method to recover transparent objects by solving an optimization function with a positionnormal consistency constraint, under the assumption of two refractions along each light path.…”