What Do Single-View 3D Reconstruction Networks Learn?

Abstract

Figure 1: We provide evidence that state-of-the-art single-view 3D reconstruction methods (AtlasNet (light green, 0.38 IoU) [12], OGN (green, 0.46 IoU) [46], Matryoshka Networks (dark green, 0.47 IoU) [37]) do not actually perform reconstruction but image classification. We explicitly design pure recognition baselines (Clustering (light blue, 0.46 IoU) and Retrieval (dark blue, 0.57 IoU)) and show that they produce similar or better results both qualitatively and quantitatively. For reference, we show the ground truth (white) and a nearest neighbor from the training set (red, 0.76 IoU). The inset shows the input image. AbstractConvolutional networks for single-view object reconstruction have shown impressive performance and have become a popular subject of research. All existing techniques are united by the idea of having an encoder-decoder network that performs non-trivial reasoning about the 3D structure of the output space. In this work, we set up two alternative approaches that perform image classification and retrieval respectively. These simple baselines yield better results than state-of-the-art methods, both qualitatively and quantitatively. We show that encoder-decoder methods are statistically indistinguishable from these baselines, thus indicating that the current state of the art in single-view object reconstruction does not actually perform reconstruction but image classification. We identify aspects of popular experimental procedures that elicit this behavior and discuss ways to improve the current state of research.

OralViewer: 3D Demonstration of Dental Surgeries for Patient Education with Oral Cavity Reconstruction from a 2D Panoramic X-ray

“…Single-view 3D reconstruction aims at generating the 3D model of an object based on a single 2D projection of it. Currently, deep Convolutional Neural Networks (ConvNets) have achieved the highest accuracy in various benchmarks by using both low-level image cues, e.g., texture, and high-level semantic information [47,53]. Our work targets at generating the voxel-based representation of teeth volumes, which estimates a voxel occupancy grid for indicating if voxels are within the space of an object.…”

“…Our task of teeth reconstruction has two unique challenges from the existing voxel-based work. (i) The reconstruction contains multiple objects (teeth) rather than a single object as in [13,47,53]. (ii) The input image of X-ray has a higher resolution than existing work (e.g., 128×128 [13]), which calls for higher computational and memory efficiency of model.…”

Deep Auxiliary Learning for Point Cloud Generation

“…According to the theory in [40], the current state-of-theart in single-view object reconstruction does not actually perform reconstruction but image classification. The convolutional layers of the encoder are identical to the corresponding parts of state-of-the-art classification model ResNet34 except that the feature map sizes are adjusted by our input size.…”

Vaccine-style-net

“…6) CD is a reasonable metric when it is used to compare two complete point cloud. However, when using it as the evaluation metric of point cloud completion task, it cannot detect the degree of the completeness, because CD is prone to outliers [1,19], and it does not possess completeness invariance when it is used to compare an incomplete point cloud with a complete point cloud. Consider the example in Fig 3, the top row of Fig 3 illustrates that the value of CD varies greatly with the position of missing part, even though the missing ratio is the same.…”