Attention-based 3D Object Reconstruction from a Single Image

Abstract

Recently, learning-based approaches for 3D reconstruction from 2D images have gained popularity due to its modern applications, e.g., 3D printers, autonomous robots, self-driving cars, virtual reality, and augmented reality. The computer vision community has applied a great effort in developing functions to reconstruct the full 3D geometry of objects and scenes. However, to extract image features, they rely on convolutional neural networks, which are ineffective in capturing long-range dependencies. In this paper, we propose to substantially improve Occupancy Networks, a state-of-the-art method for 3D object reconstruction. For such we apply the concept of self-attention within the network's encoder in order to leverage complementary input features rather than those based on local regions, helping the encoder to extract global information. With our approach, we were capable of improving the original work in 5.05% of mesh IoU, 0.83% of Normal Consistency, and more than 10× the Chamfer-L1 distance. We also perform a qualitative study that shows that our approach was able to generate much more consistent meshes, confirming its increased generalization power over the current state-of-the-art.

Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction

“…Reconstruction based on single-view images: Realistically, multi-view images from abundant object instances are difficult to acquire. For that reason, recent work like Salvi [26] introduced learning-based architectures for 3D shape reconstruction using solely single input images. Thanks to the differentiable renderer [2,18,21], several frameworks [2,7,15,18,19,21] were presented to bridge the gap between an input image and its resulting texture by utilizing differentiable rendering and image reconstruction.…”

3D Reconstruction from a Single RGB Image using Deep Learning: A Review

“…316 well. However, as [8] showed in some of their experiments, the benefit from self-attention 396 modules was highest when used at the early layers of the encoder as opposed to the later 397 layers.…”

“…-Bednarik et al [3] and Patch-Net [4] for reconstruction of depth and normal maps using a real dataset of texture-less surfaces, -HDM-Net [5] and IsMo-GAN [6], which reconstruct 3D point clouds from a synthetic dataset of textured surfaces, and -Pixel2Mesh [7], Salvi et al [8] and Yuan et al [9] for reconstruction of mesh-based models using a subset of the ShapeNet [10].…”

“…The "attentioned" ResNet-18[34] network with four self-attention blocks[37] added to it. This encoder network is used by[8] to extract image features, which are fed to a decoder with five Conditional Batch Normalization blocks followed by an occupancy function.…”

“…A smaller CD score indicates a better Chamfer-L1 (E CD 1 ): The Chamfer distance (CD) has a high computational cost for 282 meshes because of a large number of points, so an approximation called Chamfer-L1 283 is defined. It uses L1-norm instead of the Euclidean distance[8]. Smaller values are…”