A Point Set Generation Network for 3D Object Reconstruction from a Single Image

Fan, Haoqiang; Su, Hao; Guibas, Leonidas J.

doi:10.1109/cvpr.2017.264

Cited by 1,813 publications

(1,562 citation statements)

References 23 publications

(32 reference statements)

Supporting

Mentioning

1,552

Contrasting

Unclassified

Order By: Relevance

“…If Gaussian noise were used as auxiliary input, an array of Gaussian noise was feed-forwarded together with an MRI slice in the training process as follows: 10 different sets of Gaussian noise were first generated and only the "best" set (i.e., the set that yielded the lowest M * loss (Equation 1)) was used to update the DEP model's parameters. Note that this approach is similar to and inspired by Min-of-N loss in 3D object reconstruction (Fan et al, 2017) and variety loss in Social GAN (Gupta et al, 2018). In the testing process, 10 different sets of Gaussian noise were generated and the average performance was calculated.…”

Section: Experiments Setupmentioning

confidence: 99%

Automatic Spatial Estimation of White Matter Hyperintensities Evolution in Brain MRI using Disease Evolution Predictor Deep Neural Networks

Rachmadi

Valdés-Hernández²,

Makin³

et al. 2019

Preprint

View full text Add to dashboard Cite

AbstractPrevious studies have indicated that white matter hyperintensities (WMH), the main radiological feature of small vessel disease, may evolve (i.e., shrink, grow) or stay stable over a period of time. Predicting these changes are challenging because it involves some unknown clinical risk factors that leads to a non-deterministic prediction task. In this study, we propose a deep learning model to predict the evolution of WMH from baseline to follow-up (i.e., 1-year later), namely “Disease Evolution Predictor” (DEP) model, which can be adjusted to become a non-deterministic model. The DEP model receives a baseline image as input and produces a map called “Disease Evolution Map” (DEM), which represents the evolution of WMH from baseline to follow-up. Two DEP models are proposed, namely DEP-UResNet and DEP-GAN, which are representatives of the supervised (i.e., need expert-generated manual labels to generate the output) and unsupervised (i.e., do not require manual labels produced by experts) deep learning algorithms respectively. To simulate the non-deterministic and unknown parameters involved in WMH evolution, we modulate a Gaussian noise array to the DEP model as auxiliary input. This forces the DEP model to imitate a wider spectrum of alternatives in the prediction results. The alternatives of using other types of auxiliary input instead, such as baseline WMH and stroke lesion loads are also proposed and tested. Based on our experiments, the fully supervised machine learning scheme DEP-UResNet regularly performed better than the DEP-GAN which works in principle without using any expert-generated label (i.e., unsupervised). However, a semi-supervised DEP-GAN model, which uses probability maps produced by a supervised segmentation method in the learning process, yielded similar performances to the DEP-UResNet and performed best in the clinical evaluation. Furthermore, an ablation study showed that an auxiliary input, especially the Gaussian noise, improved the performance of DEP models compared to DEP models that lacked the auxiliary input regardless of the model’s architecture. To the best of our knowledge, this is the first extensive study on modelling WMH evolution using deep learning algorithms, which deals with the non-deterministic nature of WMH evolution.

show abstract

Section: Experiments Setupmentioning

confidence: 99%

Automatic Spatial Estimation of White Matter Hyperintensities Evolution in Brain MRI using Disease Evolution Predictor Deep Neural Networks

Rachmadi

Valdés-Hernández²,

Makin³

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Tulsianni et al [29] introduced ray-tracing into the picture to predict multiple semantics from an image including a 3D voxel model. Howbeit, voxel representation is known to be inefficient and computationally unfriendly [4,30]. For mesh representation, Wang et al [30] gradually deformed an elliptical mesh given an input image by using graph convolution, but mesh representation requires overhead construction, and graph convolution may result in computing redundancy as masking is needed.…”

Section: Related Workmentioning

confidence: 99%

“…In this regard, we present a novel deep method to reconstruct a 3D point cloud representation of an object from a single 2D image. Even though a point cloud representation does not possess appealing 3D geometrical properties like a mesh or CAD model, it is simple and efficient when it comes to transformation and deformation, and can produce high-quality shape models [4].…”

Section: Introductionmentioning

confidence: 99%

GraphX-Convolution for Point Cloud Deformation in 2D-to-3D Conversion

Nguyen

Choi

Kim

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

In this paper, we present a novel deep method to reconstruct a point cloud of an object from a single still image. Prior arts in the field struggle to reconstruct an accurate and scalable 3D model due to either the inefficient and expensive 3D representations, the dependency between the output and number of model parameters or the lack of a suitable computing operation. We propose to overcome these by deforming a random point cloud to the object shape through two steps: feature blending and deformation. In the first step, the global and point-specific shape features extracted from a 2D object image are blended with the encoded feature of a randomly generated point cloud, and then this mixture is sent to the deformation step to produce the final representative point set of the object. In the deformation process, we introduce a new layer termed as GraphX that considers the inter-relationship between points like common graph convolutions but operates on unordered sets. Moreover, with a simple trick, the proposed model can generate an arbitrary-sized point cloud, which is the first deep method to do so. Extensive experiments verify that we outperform existing models and halve the state-of-the-art distance score in single image 3D reconstruction.

show abstract

“…However, PointNet is not a generative model of point sets, but rather it maps input point sets to output such as a model classification, or part segmentation. In related work, a conditional generative model of unordered point sets was introduced in [FSG16], where given an image, a collection of 3D output points was synthesized that captures the coarse shape of objects in the image. The closest work to ours is Huang et al .…”

Section: Related Workmentioning

confidence: 99%

The shape variational autoencoder: A deep generative model of part‐segmented 3D objects

Nash

Williams

2017

Computer Graphics Forum

136

View full text Add to dashboard Cite

We introduce a generative model of part‐segmented 3D objects: the shape variational auto‐encoder (ShapeVAE). The ShapeVAE describes a joint distribution over the existence of object parts, the locations of a dense set of surface points, and over surface normals associated with these points. Our model makes use of a deep encoder‐decoder architecture that leverages the part‐decomposability of 3D objects to embed high‐dimensional shape representations and sample novel instances. Given an input collection of part‐segmented objects with dense point correspondences the ShapeVAE is capable of synthesizing novel, realistic shapes, and by performing conditional inference enables imputation of missing parts or surface normals. In addition, by generating both points and surface normals, our model allows for the use of powerful surface‐reconstruction methods for mesh synthesis. We provide a quantitative evaluation of the ShapeVAE on shape‐completion and test‐set log‐likelihood tasks and demonstrate that the model performs favourably against strong baselines. We demonstrate qualitatively that the ShapeVAE produces plausible shape samples, and that it captures a semantically meaningful shape‐embedding. In addition we show that the ShapeVAE facilitates mesh reconstruction by sampling consistent surface normals.

show abstract

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

Cited by 1,813 publications

References 23 publications

Automatic Spatial Estimation of White Matter Hyperintensities Evolution in Brain MRI using Disease Evolution Predictor Deep Neural Networks

Automatic Spatial Estimation of White Matter Hyperintensities Evolution in Brain MRI using Disease Evolution Predictor Deep Neural Networks

GraphX-Convolution for Point Cloud Deformation in 2D-to-3D Conversion

The shape variational autoencoder: A deep generative model of part‐segmented 3D objects

Contact Info

Product

Resources

About