Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

Sun, Xiaobin; Wu, Jiajun; Zhang, Xiuming; Zhang, Zhoutong; Zhang, Chengkai; Xue, Tao; Tenenbaum, Joshua B.; Freeman, William T.

doi:10.1109/cvpr.2018.00314

Cited by 369 publications

(382 citation statements)

References 61 publications

(92 reference statements)

Supporting

Mentioning

381

Contrasting

Unclassified

Order By: Relevance

“…Reconstructing real-world objects. To qualitatively evaluate the generalization performance of our method on the real images, we test our network on the Pix3D [28] dataset by using the model trained on the ShapeNet [3]. Figure 6 shows the results reconstructed by our method and AtlasNet, where the objects in the images are manually segmented.…”

Section: Comparisons With the State-of-the-artsmentioning

confidence: 99%

Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks

Pan

Han

Chen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

171

167

View full text Add to dashboard Cite

Reconstructing the 3D mesh of a general object from a single image is now possible thanks to the latest advances of deep learning technologies. However, due to the nontrivial difficulty of generating a feasible mesh structure, the stateof-the-art approaches [16,32] often simplify the problem by learning the displacements of a template mesh that deforms it to the target surface. Though reconstructing a 3D shape with complex topology can be achieved by deforming multiple mesh patches, it remains difficult to stitch the results to ensure a high meshing quality. In this paper, we present an end-to-end single-view mesh reconstruction framework that is able to generate high-quality meshes with complex topologies from a single genus-0 template mesh. The key to our approach is a novel progressive shaping framework that alternates between mesh deformation and topology modification. While a deformation network predicts the per-vertex translations that reduce the gap between the reconstructed mesh and the ground truth, a novel topology modification network is employed to prune the error-prone faces, enabling the evolution of topology. By iterating over the two procedures, one can progressively modify the mesh topology while achieving higher reconstruction accuracy. Moreover, a boundary refinement network is designed to refine the boundary conditions to further improve the visual quality of the reconstructed mesh. Extensive experiments demonstrate that our approach outperforms the current state-of-the-art methods both qualitatively and quantitatively, especially for the shapes with complex topologies.

show abstract

Section: Comparisons With the State-of-the-artsmentioning

confidence: 99%

Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks

Pan

Han

Chen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

171

167

View full text Add to dashboard Cite

show abstract

“…To make it easier for PCDNet to serve as a baseline in subsequent research, we reported two common metric scores which are CD and intersection over union (IoU). CD is our main criterion, not because PCDNet is trained using CD, but it is better correlated with human perception [27]. IoU quantifies the overlapping region between two input sets.…”

Section: Resultsmentioning

confidence: 99%

GraphX-Convolution for Point Cloud Deformation in 2D-to-3D Conversion

Nguyen

Choi

Kim

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

In this paper, we present a novel deep method to reconstruct a point cloud of an object from a single still image. Prior arts in the field struggle to reconstruct an accurate and scalable 3D model due to either the inefficient and expensive 3D representations, the dependency between the output and number of model parameters or the lack of a suitable computing operation. We propose to overcome these by deforming a random point cloud to the object shape through two steps: feature blending and deformation. In the first step, the global and point-specific shape features extracted from a 2D object image are blended with the encoded feature of a randomly generated point cloud, and then this mixture is sent to the deformation step to produce the final representative point set of the object. In the deformation process, we introduce a new layer termed as GraphX that considers the inter-relationship between points like common graph convolutions but operates on unordered sets. Moreover, with a simple trick, the proposed model can generate an arbitrary-sized point cloud, which is the first deep method to do so. Extensive experiments verify that we outperform existing models and halve the state-of-the-art distance score in single image 3D reconstruction.

show abstract

“…Tatarchenko et al [79], Lin et al [69], and Sun et al [10] also estimate the binary/silhouette masks, along with the depth maps. The binary masks have been used to filter out points that are not backprojected to the surface in 3D space.…”

Section: Intermediatingmentioning

confidence: 99%

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Han

Laga

Bennamoun

2021

IEEE Trans. Pattern Anal. Mach. Intell.

272

136

View full text Add to dashboard Cite

3D reconstruction is a longstanding ill-posed problem, which has been explored for decades by the computer vision, computer graphics, and machine learning communities. Since 2015, image-based 3D reconstruction using convolutional neural networks (CNN) has attracted increasing interest and demonstrated an impressive performance. Given this new era of rapid evolution, this article provides a comprehensive survey of the recent developments in this field. We focus on the works which use deep learning techniques to estimate the 3D shape of generic objects either from a single or multiple RGB images. We organize the literature based on the shape representations, the network architectures, and the training mechanisms they use. While this survey is intended for methods which reconstruct generic objects, we also review some of the recent works which focus on specific object classes such as human body shapes and faces. We provide an analysis and comparison of the performance of some key papers, summarize some of the open problems in this field, and discuss promising directions for future research.

show abstract

Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

Cited by 369 publications

References 61 publications

Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks

Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks

GraphX-Convolution for Point Cloud Deformation in 2D-to-3D Conversion

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Contact Info

Product

Resources

About