Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion

Pavllo, Dario; Tan, David Joseph; Rakotosaona, Marie-Julie; Tombari, Federico

doi:10.48550/arxiv.2211.11674

Cited by 3 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PSNR ↑ SSIM ↑ LPIPS ↓ NeRF from image [59] 16.26 0.614 0.456 PixelNeRF [4] 20.87 0.714 0.396 SinNeRF [58] 20.13 0.688 0.417 RealFusion [28] 19.62 0.629 0.426 PIFu-HD [39] 18.17 0.638 0.439 Magic123 [60] 21.68 0.716 0.390 Our method 23.21 0.769 0.373…”

Section: Methodsmentioning

confidence: 99%

“…In the quantitative evaluation, we conduct experiments on DeepFashion 3D dataset. We compare our proposed method with 6 state-of-the-art single-image to neural representation algorithms: PixelNeRF [4], SinNeRF [58], NeRF from image [59] RealFusion [28], PIFu-HD [39] and Magic123 [60]. In SinNeRF, we used ground truth depth values for training.…”

Section: Quantitative Evaluationmentioning

confidence: 99%

“…We can observe that each of these algorithms has its own limitations. Diffusion-based methods like NeRF from image [59], RealFusion [28] and Magic123 [60] suffered from color differences between the input images and the reconstructed 3D models. This issue could be attributed to the image inversion used in these algorithms The output results of PixelNeRF [4] and PIFu-HD [39] exhibit blurriness in views.…”

Section: Inputmentioning

confidence: 99%

See 2 more Smart Citations

Single-View 3D Garment Reconstruction Using Neural Volumetric Rendering

Chen,

Xie,

Yang

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Reconstructing 3D garment models usually requires laborious data-fetching processes, such as expensive lidar, multiple-view images, or SMPL models of the garments. In this paper, we propose a neat framework that takes single-image inputs for generating pseudo-sparse views of 3D garments and synthesizing multi-view images into a high-quality 3D neural model. Specifically, our framework combines a pretrained pseudo sparse view generator and a volumetric signed distance function (SDF) representationbased network for 3D garment modeling, which uses neural networks to represent both the density and radiance fields. We further introduce a stride fusion strategy to minimize the pixel-level loss in key viewpoints and semantic loss in random viewpoints, which produces view-consistent geometry and sharp texture details. Finally, a multi-view rendering module utilizes the learned SDF representation to generate multi-view garment images and extract accurate mesh and texture from them. We evaluate our proposed framework on the Deep Fashion 3D dataset and achieve state-of-the-art performance in terms of both quantitative and qualitative evaluations.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Quantitative Evaluationmentioning

confidence: 99%

Section: Inputmentioning

confidence: 99%

See 1 more Smart Citation

Single-View 3D Garment Reconstruction Using Neural Volumetric Rendering

Chen,

Xie,

Yang

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Category-level methods learn the shared shape prior within a category and thus eliminate the need for instance-level mesh models at test time (Wang et al 2019;Tian, Jr., and Lee 2020;Lee et al 2021;Wang, Chen, and Dou 2021;Chen, Li, and Xu 2020;Chen and Dou 2021;Pavllo et al 2023). Most of these approaches try to infer correspondences from pixels to 3D points in a Normalized Object Coordinate Space (NOCS).…”

Section: Related Workmentioning

confidence: 99%

MFOS: Model-Free & One-Shot Object Pose Estimation

Lee,

Cabon,

Brégier

et al. 2024

AAAI

View full text Add to dashboard Cite

Existing learning-based methods for object pose estimation in RGB images are mostly model-specific or category based. They lack the capability to generalize to new object categories at test time, hence severely hindering their practicability and scalability. Notably, recent attempts have been made to solve this issue, but they still require accurate 3D data of the object surface at both train and test time. In this paper, we introduce a novel approach that can estimate in a single forward pass the pose of objects never seen during training, given minimum input. In contrast to existing state-of-the-art approaches, which rely on task-specific modules, our proposed model is entirely based on a transformer architecture, which can benefit from recently proposed 3D-geometry general pretraining. We conduct extensive experiments and report state-of-the-art one-shot performance on the challenging LINEMOD benchmark. Finally, extensive ablations allow us to determine good practices with this relatively new type of architecture in the field.

show abstract

“…While most of the research in this field focuses on shape reconstruction, a handful of methods adopt implicit fields to estimate object pose [26,1]. A straightforward way is to jointly reconstruct the object surface and estimate its pose [3,25,16] with a unified framework. For example, ShAPO [13] jointly predicts object shape, pose, and size in a single-shot manner.…”

Section: Implicit Field For Pose Estimationmentioning

confidence: 99%

SOCS: Semantically-aware Object Coordinate Space for Category-Level 6D Object Pose Estimation under Large Shape Variations

Wan¹,

Shi²,

Xu³

2023

Preprint

View full text Add to dashboard Cite

Most learning-based approaches to category-level 6D pose estimation are design around normalized object coordinate space (NOCS). While being successful, NOCS-based methods become inaccurate and less robust when handling objects of a category containing significant intra-category shape variations. This is because the object coordinates induced by global and rigid alignment of objects are semantically incoherent, making the coordinate regression hard to learn and generalize. We propose Semantically-aware Object Coordinate Space (SOCS) built by warping-andaligning the objects guided by a sparse set of keypoints with semantically meaningful correspondence. SOCS is semantically coherent: Any point on the surface of a object can be mapped to a semantically meaningful location in SOCS, allowing for accurate pose and size estimation under large shape variations. To learn effective coordinate regression to SOCS, we propose a novel multi-scale coordinate-based attention network. Evaluations demonstrate that our method is easy to train, well-generalizing for large intra-category shape variations and robust to inter-object occlusions.

show abstract

Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion

Cited by 3 publications

References 0 publications

Single-View 3D Garment Reconstruction Using Neural Volumetric Rendering

Single-View 3D Garment Reconstruction Using Neural Volumetric Rendering

MFOS: Model-Free & One-Shot Object Pose Estimation

SOCS: Semantically-aware Object Coordinate Space for Category-Level 6D Object Pose Estimation under Large Shape Variations

Contact Info

Product

Resources

About