Deformation-Aware 3D Model Embedding and Retrieval

Uy, Mikaela Angelina; Huang, Jingwei; Sung, Minhyuk; Birdal, Tolga; Guibas, Leonidas J.

doi:10.1007/978-3-030-58571-6_24

Cited by 30 publications

(27 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3. For the Photoshapes dataset, Ed-itNeRF is trained using 600 instances with 40 We show textand-exemplar driven shape editing results of our method and the baseline method without using our disentangled technique. When editing the shape, the latter can change the appearance, while ours keeps the appearance unchanged.…”

Section: Compared To Editnerfmentioning

confidence: 99%

“…Editing NeRF (e.g., deforming the shape or changing the appearance color), however, is an extremely challenging task. First, since NeRF is an implicit function optimized per scene, we cannot directly edit the shape using the intuitively tools for the explicit representations [35,42,41,40]. Second, unlike image manipulation where the single-view information is enough to guide the editing [19,43,44], the multi-view dependency of NeRF makes the manipulation way more difficult to control without the multi-view information.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

Wang¹,

Chai²,

He³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present CLIP-NeRF, a multi-modal 3D object manipulation method for neural radiance fields (NeRF). By leveraging the joint language-image embedding space of the recent Contrastive Language-Image Pre-Training (CLIP) model, we propose a unified framework that allows manipulating NeRF in a user-friendly way, using either a short text prompt or an exemplar image. Specifically, to combine the novel view synthesis capability of NeRF and the controllable manipulation ability of latent representations from generative models, we introduce a disentangled conditional NeRF architecture that allows individual control over both shape and appearance. This is achieved by performing the shape conditioning via applying a learned deformation field to the positional encoding and deferring color conditioning to the volumetric rendering stage. To bridge this disentangled latent representation to the CLIP embedding, we design two code mappers that take a CLIP embedding as input and update the latent codes to reflect the targeted editing. The mappers are trained with a CLIP-based matching loss to ensure the manipulation accuracy. Furthermore, we propose an inverse optimization method that accurately projects an input image to the latent codes for manipulation to enable editing on real images. We evaluate our approach by extensive experiments on a variety of text prompts and exemplar images and also provide an intuitive interface for interactive editing. Our implementation is available at https://cassiepython.github.io/ clipnerf/

show abstract

Section: Compared To Editnerfmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

Wang¹,

Chai²,

He³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Reconstruction based on retrieval and deformation: To generate 3D shape such that it resembles the target shape as close as possible, previous work [30,31,34] combined retrieval with deformation for 3D shape reconstruction. Given a query, e.g.…”

Section: Related Workmentioning

confidence: 99%

Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction

Lin

Tian

2021

Preprint

View full text Add to dashboard Cite

Recently, learning-based approaches for 3D model reconstruction have attracted attention owing to its modern applications such as Extended Reality(XR), robotics and self-driving cars. Several approaches presented good performance on reconstructing 3D shapes by learning solely from images, i.e., without using 3D models in training. Challenges, however, remain in texture generation due to the gap between 2D and 3D modals. In previous work, the grid sampling mechanism from Spatial Transformer Networks was adopted to sample color from an input image to formulate texture. Despite its success, the existing framework has limitations on searching scope in sampling, resulting in flaws in generated texture and consequentially on rendered 3D models. In this paper, to solve that issue, we present a novel sampling algorithm by optimizing the gradient of predicted coordinates based on the variance on the sampling image. Taking into account the semantics of the image, we adopt Fréchet Inception Distance (FID) to form a loss function in learning, which helps bridging the gap between rendered images and input images. As a result, we greatly improve generated texture. Furthermore, to optimize 3D shape reconstruction and to accelerate convergence at training, we adopt part segmentation and template learning in our model. Without any 3D supervision in learning, and with only a collection of single-view 2D images, the shape and texture learned by our model outperform those from previous work. We demonstrate the performance with experimental results on a publically available dataset.

show abstract

“…Most existing work focuses either on category-level object retrieval or object retrieval with fixed poses [4]- [6] or pose estimation between identical object pairs [7]- [9], differing only due to noise or occlusion but not due to shape variation. This paper considers object pose estimation from partial point-cloud observations to enable online object-level mapping.…”

Section: Introductionmentioning

confidence: 99%

CORSAIR: Convolutional Object Retrieval and Symmetry-AIded Registration

Zhao

Feng

Jadhav

et al. 2021

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

This paper considers online object-level mapping using partial point-cloud observations obtained online in an unknown environment. We develop and approach for fully Convolutional Object Retrieval and Symmetry-AIded Registration (CORSAIR). Our model extends the Fully Convolutional Geometric Features model to learn a global object-shape embedding in addition to local point-wise features from the point-cloud observations. The global feature is used to retrieve a similar object from a category database, and the local features are used for robust pose registration between the observed and the retrieved object. Our formulation also leverages symmetries, present in the object shapes, to obtain promising local-feature pairs from different symmetry classes for matching. We present results from synthetic and real-world datasets with different object categories to verify the robustness of our method.

show abstract

Deformation-Aware 3D Model Embedding and Retrieval

Cited by 30 publications

References 45 publications

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction

CORSAIR: Convolutional Object Retrieval and Symmetry-AIded Registration

Contact Info

Product

Resources

About