Single image 3D object reconstruction based on deep learning: A review

Fu, Kui; Peng, Jiansheng; He, Qiang; Zhang, Hanxiao

doi:10.1007/s11042-020-09722-8

Cited by 84 publications

(44 citation statements)

References 125 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the future, images with textures and backgrounds can be used for rendering to enrich the dataset, which will make the model more robust to 3D object reconstruction from 2D images in real scenes. In addition, there are other methods, such as exploring new algorithms to extract more effective image features, using different training architectures, and supervising methods to optimize [54].…”

Section: Methodsmentioning

confidence: 99%

Improved Multiview Decomposition for Single-Image High-Resolution 3D Object Reconstruction

Peng

Wei

et al. 2020

Wireless Communications and Mobile Computing

Self Cite

View full text Add to dashboard Cite

As a representative technology of artificial intelligence, 3D reconstruction based on deep learning can be integrated into the edge computing framework to form an intelligent edge and then realize the intelligent processing of the edge. Recently, high-resolution representation of 3D objects using multiview decomposition (MVD) architecture is a fast reconstruction method for generating objects with realistic details from a single RGB image. The results of high-resolution 3D object reconstruction are related to two aspects. On the one hand, a low-resolution reconstruction network represents a good 3D object from a single RGB image. On the other hand, a high-resolution reconstruction network maximizes fine low-resolution 3D objects. To improve these two aspects and further enhance the high-resolution reconstruction capabilities of the 3D object generation network, we study and improve the low-resolution 3D generation network and the depth map superresolution network. Eventually, we get an improved multiview decomposition (IMVD) network. First, we use a 2D image encoder with multifeature fusion (MFF) to enhance the feature extraction capability of the model. Second, a 3D decoder using an effective subpixel convolutional neural network (3D ESPCN) improves the decoding speed in the decoding stage. Moreover, we design a multiresidual dense block (MRDB) to optimize the depth map superresolution network, which allows the model to capture more object details and reduce the model parameters by approximately 25% when the number of network layers is doubled. The experimental results show that the proposed IMVD is better than the original MVD in the 3D object superresolution experiment and the high-resolution 3D reconstruction experiment of a single image.

show abstract

Section: Methodsmentioning

confidence: 99%

Improved Multiview Decomposition for Single-Image High-Resolution 3D Object Reconstruction

Peng

Wei

et al. 2020

Wireless Communications and Mobile Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…Ma and Karaman [25] added sparse depth samples from point cloud data to the network, which improved the accuracy by more than 50%. Aside from depth maps, the output of CNNs has variable formats, such as mesh polygon and point clouds, to describe 3D information when using a single image as the input [45]. Fan et al [46] and Xia et al [47] proposed methods that output a group of point coordinates to describe 3D information.…”

Section: Related Workmentioning

confidence: 99%

Mars3DNet: CNN-Based High-Resolution 3D Reconstruction of the Martian Surface from Single Images

Chen

Liu

2021

Remote Sensing

View full text Add to dashboard Cite

Three-dimensional (3D) surface models, e.g., digital elevation models (DEMs), are important for planetary exploration missions and scientific research. Current DEMs of the Martian surface are mainly generated by laser altimetry or photogrammetry, which have respective limitations. Laser altimetry cannot produce high-resolution DEMs; photogrammetry requires stereo images, but high-resolution stereo images of Mars are rare. An alternative is the convolutional neural network (CNN) technique, which implicitly learns features by assigning corresponding inputs and outputs. In recent years, CNNs have exhibited promising performance in the 3D reconstruction of close-range scenes. In this paper, we present a CNN-based algorithm that is capable of generating DEMs from single images; the DEMs have the same resolutions as the input images. An existing low-resolution DEM is used to provide global information. Synthetic and real data, including context camera (CTX) images and DEMs from stereo High-Resolution Imaging Science Experiment (HiRISE) images, are used as training data. The performance of the proposed method is evaluated using single CTX images of representative landforms on Mars, and the generated DEMs are compared with those obtained from stereo HiRISE images. The experimental results show promising performance of the proposed method. The topographic details are well reconstructed, and the geometric accuracies achieve root-mean-square error (RMSE) values ranging from 2.1 m to 12.2 m (approximately 0.5 to 2 pixels in the image space). The experimental results show that the proposed CNN-based method has great potential for 3D surface reconstruction in planetary applications.

show abstract

“…Thus, our technique could be useful for mesh-based reconstruction with a sensor of this kind. Finally, one can rely on the ability of a deep neural network to reconstruct a 3D shape from a single image (or multiple images) [ 30 , 31 ]. In this area, the conclusion is interestingly that a mesh-based method can generate 3D shapes with higher quality than voxel-based and point-based methods.…”

Section: Related Workmentioning

confidence: 99%

Surface Reconstruction from Structured Light Images Using Differentiable Rendering

Jensen

Hannemose

Bærentzen

et al. 2021

Sensors

View full text Add to dashboard Cite

When 3D scanning objects, the objective is usually to obtain a continuous surface. However, most surface scanning methods, such as structured light scanning, yield a point cloud. Obtaining a continuous surface from a point cloud requires a subsequent surface reconstruction step, which is directly affected by any error from the computation of the point cloud. In this work, we propose a one-step approach in which we compute the surface directly from structured light images. Our method minimizes the least-squares error between photographs and renderings of a triangle mesh, where the vertex positions of the mesh are the parameters of the minimization problem. To ensure fast iterations during optimization, we use differentiable rendering, which computes images and gradients in a single pass. We present simulation experiments demonstrating that our method for computing a triangle mesh has several advantages over approaches that rely on an intermediate point cloud. Our method can produce accurate reconstructions when initializing the optimization from a sphere. We also show that our method is good at reconstructing sharp edges and that it is robust with respect to image noise. In addition, our method can improve the output from other reconstruction algorithms if we use these for initialization.

show abstract

Single image 3D object reconstruction based on deep learning: A review

Cited by 84 publications

References 125 publications

Improved Multiview Decomposition for Single-Image High-Resolution 3D Object Reconstruction

Improved Multiview Decomposition for Single-Image High-Resolution 3D Object Reconstruction

Mars3DNet: CNN-Based High-Resolution 3D Reconstruction of the Martian Surface from Single Images

Surface Reconstruction from Structured Light Images Using Differentiable Rendering

Contact Info

Product

Resources

About