On the generalization of learning-based 3D reconstruction

Abstract

State-of-the-art learning-based monocular 3D reconstruction methods learn priors over object categories on the training set, and as a result struggle to achieve reasonable generalization to object categories unseen during training. In this paper we study the inductive biases encoded in the model architecture that impact the generalization of learning-based 3D reconstruction methods. We find that 3 inductive biases impact performance: the spatial extent of the encoder, the use of the underlying geometry of the scene to describe point features, and the mechanism to aggregate information from multiple views. Additionally, we propose mechanisms to enforce those inductive biases: a point representation that is aware of camera position, and a variance cost to aggregate information across views. Our model achieves state-of-the-art results on the standard ShapeNet 3D reconstruction benchmark in various settings.

Deep Learning for 3D Reconstruction, Augmentation, and Registration: A Review Paper

“…3D34D [ 17 ]: The authors of this model employ a UNet encoder, producing feature maps to produce geometry-aware point representations of object categories unseen during training. For 3D object reconstruction, this study employs multi-view images with ground truth camera postures and pixel-aligned feature representations.…”

“…Cycle-consistency-based approach [15] ShapeNet [10], Pix3D [16] Point Cloud 3D34D [17] ShapeNet [10] Point Cloud Unsupervised learning of 3D structure [18] ShapeNet [10], MNIST3D [19] Point Cloud…”

“…3D34D [17] Obtains a more expressive intermediate shape representation by locally assigning features and 3D points.…”

Fostering Generalization in Single-view 3D Reconstruction by Learning a Hierarchy of Local and Global Shape Priors

“…Similar in spirit to our approach, Chibane et al [6] propose to extract a hierarchy of features for solving several 3D-to-3D tasks. Bautista et al [3] locally assign features and 3D points in order to get a more expressive intermediate shape representation. Most similar to ours is the work from Genova et al [11].…”

“…Most existing approaches are encoder-decoder networks [7,10,12,19,27,30,34] and have been shown to barely generalize to novel shape categories [38]. Only few works have targeted generalization explicitly [3,32,38]. They argue that, for better generalization, the problem should be split into two parts: (1) prediction of a geometric representation of the visible parts from a single RGB image and (2) prediction of the final shape from the geometric representation.…”