Implicit Mesh Reconstruction from Unannotated Image Collections

Tulsiani, Shubham; Kulkarni, Nilesh; Gupta, Abhinav

doi:10.48550/arxiv.2007.08504

Cited by 15 publications

(40 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the advances in differentiable rendering (Kato et al, 2018;Laine et al, 2020;, these have also been leveraged in learning based frameworks for shape prediction (Kanazawa et al, 2018;Gkioxari et al, 2019;Goel et al, 2020) and view synthesis (Riegler and Koltun, 2020). Whereas these approaches use an explicit discrete mesh, some recent methods have proposed using continuous neural surface parametrization like ours to represent shape (Groueix et al, 2018) and texture (Tulsiani et al, 2020;Bhattad et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild

Zhang,

Yang,

Tulsiani

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent history has seen a tremendous growth of work exploring implicit representations of geometry and radiance, popularized through Neural Radiance Fields (NeRF). Such works are fundamentally based on a (implicit) volumetric representation of occupancy, allowing them to model diverse scene structure including translucent objects and atmospheric obscurants. But because the vast majority of real-world scenes are composed of well-defined surfaces, we introduce a surface analog of such implicit models called Neural Reflectance Surfaces (NeRS). NeRS learns a neural shape representation of a closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions. Even more importantly, surface parameterizations allow NeRS to learn (neural) bidirectional surface reflectance functions (BRDFs) that factorize view-dependent appearance into environmental illumination, diffuse color (albedo), and specular "shininess." Finally, rather than illustrating our results on synthetic scenes or controlled in-the-lab capture, we assemble a novel dataset of multi-view images from online marketplaces for selling goods. Such "in-the-wild" multi-view image sets pose a number of challenges, including a small number of views with unknown/rough camera estimates. We demonstrate that surface-based neural reconstructions enable learning from such data, outperforming volumetric neural rendering-based reconstructions. We hope that NeRS serves as a first step toward building scalable, high-quality libraries of real-world shape, materials, and illumination. The project page with code and video visualizations can be found at jasonyzhang.com/ners.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Inspired by Groueix et al (2018); Tulsiani et al (2020), we address these challenges by adopting a continuous surface representation via a neural network. We illustrate this representation in Fig.…”

Section: Camera Surfacementioning

confidence: 99%

NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild

Zhang,

Yang,

Tulsiani

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…3D Deformable Mesh Representations. Our algorithm is also related to methods [13,14,15,5] that represent instance shapes as mesh deformations of a mesh primitive, i.e. a sphere template.…”

Section: Related Workmentioning

confidence: 99%

“…In this work, we introduce a novel canonical space where dense (i.e., point-level) correspondences for all the shapes of a category can be explicitly obtained from. Inspired by 3D mesh representation [13,14,15,5] where shapes from one category are represented as deformations on top of a shape primitive, in our work, we set the canonical space as a 3D UV sphere. Our goal is to learn a "point cloud-to-sphere mapping" such that corresponding parts from different instances overlap when mapped onto the canonical sphere.…”

Section: Introductionmentioning

confidence: 99%

Learning 3D Dense Correspondence via Canonical Point Autoencoder

Cheng,

Li,

Sun

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a canonical point autoencoder (CPAE) that predicts dense correspondences between 3D shapes of the same category. The autoencoder performs two key functions: (a) encoding an arbitrarily ordered point cloud to a canonical primitive, e.g., a sphere, and (b) decoding the primitive back to the original input instance shape. As being placed in the bottleneck, this primitive plays a key role to map all the unordered point clouds on the canonical surface and to be reconstructed in an ordered fashion. Once trained, points from different shape instances that are mapped to the same locations on the primitive surface are determined to be a pair of correspondence. Our method does not require any form of annotation or self-supervised part segmentation network and can handle unaligned input point clouds. Experimental results on 3D semantic keypoint transfer and part segmentation transfer show that our model performs favorably against state-of-the-art correspondence learning methods.Preprint. Under review.

show abstract

“…Early methods employ inflation techniques to extract shapes from silhouettes [36,49]. More recent methods learn end-to-end predictors, such as CMR [25], U-CMR [17] and IMR [48], to produce textured 3D animal mesh from a single image. So far, the outputs of these methods have limited realism and cannot be re-animated.…”

Section: Related Workmentioning

confidence: 99%