Learning Shape Priors for Single-View 3D Completion And Reconstruction

Abstract

The problem of single-view 3D shape completion or reconstruction is challenging, because among the many possible shapes that explain an observation, most are implausible and do not correspond to natural objects. Recent research in the field has tackled this problem by exploiting the expressiveness of deep convolutional networks. In fact, there is another level of ambiguity that is often overlooked: among plausible shapes, there are still multiple shapes that fit the 2D image equally well; i.e., the ground truth shape is non-deterministic given a single-view input. Existing fully supervised approaches fail to address this issue, and often produce blurry mean shapes with smooth surfaces but no fine details. In this paper, we propose ShapeHD, pushing the limit of single-view shape completion and reconstruction by integrating deep generative models with adversarially learned shape priors. The learned priors serve as a regularizer, penalizing the model only if its output is unrealistic, not if it deviates from the ground truth. Our design thus overcomes both levels of ambiguity aforementioned. Experiments demonstrate that ShapeHD outperforms state of the art by a large margin in both shape completion and shape reconstruction on multiple real datasets.

Efficient inverse graphics in biological face processing

“…With other collaborators, we have recently shown in an AI context that EIG-like networks for efficient inference of 3D shapes from 2D images via 2.5D sketches can work for arbitrary object classes (e.g., chairs, cars) 29,58 , and can even generalize to a range of novel, unseen classes 59 . In future work, we hope to explore these models of how the ventral visual pathway processes other object classes with functionally specific, localized representations (bodies, hands, word forms), as well as objects more generally.…”

Few-Shot Generalization for Single-Image 3D Reconstruction via Priors

“…The notion of using or learning priors has also been explored before. One approach to using priors is to use an adversary to enforce realistic reconstruction [28,12]. Cherabier et al use shape priors to learn from relatively little data, but focus on scene reconstruction with semantically labeled depth maps as inputs [3].…”

Breaking Medical Data Sharing Boundaries by Employing Artificial Radiographs

“…The concepts demonstrated in this work rely on two-dimensional images, but there is no principal restriction on the number of dimensions that the real and artificial images are allowed to have, or even that the data has to consist of images only [38]. Thus, the same concept could be translated to volumetric computed tomography or magnetic resonance images, to fluoroscopy, to time-series of volumetric data (e.g.…”