Shape-from-Template

Abstract

We study a problem that we call Shape-from-Template, which is the problem of reconstructing the shape of a deformable surface from a single image and a 3D template. Current methods in the literature address the case of isometric deformations, and relax the isometry constraint to the convex inextensibility constraint, solved using the so-called maximum depth heuristic. We call these methods zeroth-order since they use image point locations (the zeroth-order differential structure) to solve the shape inference problem from a perspective image. We propose a novel class of methods that we call first-order. The key idea is to use both image point locations and their first-order differential structure. The latter can be easily extracted from a warp between the template and the input image. We give a unified problem formulation as a system of PDEs for isometric and conformal surfaces that we solve analytically. This has important consequences. First, it gives the first analytical algorithms to solve this type of reconstruction problems. Second, it gives the first algorithms to solve for the exact constraints. Third, it allows us to study the well-posedness of this type of reconstruction: we establish that isometric surfaces can be reconstructed unambiguously and that conformal surfaces can be reconstructed up to a few discrete ambiguities and a global scale. In the latter case, the candidate solution surfaces are obtained analytically. Experimental results on simulated and real data show that our isometric methods generally perform as well as or outperform state of the art approaches in terms of reconstruction accuracy, while our conformal methods largely outperform all isometric methods for extensible deformations.

Variational Uncalibrated Photometric Stereo Under General Lighting

“…Many approaches to the problem of 3D-reconstruction from photographies have been studied, and they are grouped under the generic naming "shape-from-X", where X stands for the clue which is being used (shadows [44], contours [10], texture [49], template [6], structured light [16], motion [35], focus [36], silhouettes [21], etc.). Geometric shape-from-X techniques are based on the identification and analysis of feature point or areas in the image.…”

“…Figure 2 for a visualization). Plugging (6) and specifics of spherical harmonics [8] into (4), we finalize our image formation model as:…”

Reduced order modeling for physically-based augmented reality

“…An impressive effort of research has been devoted to solving the so-called Non-Rigid Structure from Motion (NRSfM) problem. A non-exhaustive list of techniques in this field could consider the Shape from Template (SfT) techniques [8,9], an Extended Kalman Filter approach employing linear elastic plate finite element to simulate the visible surface of the solid [6,31], or recent low-rank approaches to the that initialize the sequence with rigid motion only and sequentially improve the basis of the method [32], to name a few.…”

“…If one or more objects in the real, physical scene is deformable, the problem turns out to be ill posed, although good solutions exist under different assumptions [5]. Non-Rigid Structure from Motion, NRSfM, [6,7] and Shape from Template, SfT, methods [8,9] are some good examples of different approaches to this problem.…”

Model-free Consensus Maximization for Non-Rigid Shapes

“…The first problem is that of maximizing consensus between matched 3D surface points in non-rigid 3D shapes using the isometry prior. Isometry is a widely used prior in registration [14,15,[20][21][22] as well as 3D reconstruction [23,24]. Most non-rigid shape registration methods [15,[20][21][22] start with a 3D descriptor such as the SHOT descriptor [25] or heat kernels [26] and establish correspondences between shapes through energy minimization.…”