DiffRF: Rendering-Guided 3D Radiance Field Diffusion

Muller, Norman; Siddiqui, Yawar; Porzi, Lorenzo; Bulò, Samuel Rota; Kontschieder, Peter; Nießner, Matthias

doi:10.48550/arxiv.2212.01206

Cited by 2 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the impact of diffusion-based models has transcended into the realm of 3D tasks. Notably, applications such as 3D generation [4], 3D reconstruction by single image [1], [18], point clouds [32], [59] or voxel grids [63], text-to-3D generation [4], [11], [28], [44], radiance fields generation [38], [54] have experienced notable progress due to adopting diffusion processes. Between these, the recently proposed DiffRF [38] provides the first direct synthesis of volumetric radiance fields using a diffusion-based generative model.…”

Section: Diffusion Modelsmentioning

confidence: 99%

“…Notably, applications such as 3D generation [4], 3D reconstruction by single image [1], [18], point clouds [32], [59] or voxel grids [63], text-to-3D generation [4], [11], [28], [44], radiance fields generation [38], [54] have experienced notable progress due to adopting diffusion processes. Between these, the recently proposed DiffRF [38] provides the first direct synthesis of volumetric radiance fields using a diffusion-based generative model. The model is optimized by pairing a denoising formulation with rendering loss between images after rendering with approximated time step values.…”

Section: Diffusion Modelsmentioning

confidence: 99%

“…Radiance fields represent a 3D object in an implicit way in terms of (ξ, σ), where RGB color field ξ : X → R 3 and density field σ : X → R + [38]. Specifically in NeRF [34], the spatial coordinates p = (x, y, z) and view direction v = (θ, φ) of a sampled point on a ray is fed through the learned MLP network to obtain the radiance field values.…”

Section: ) Radiance Fieldmentioning

confidence: 99%

“…As mentioned in Section III-B2, obtaining ground truth radiance fields is non-trivial. The challenge is addressed by utilizing a rendering loss, comparing images obtained by each radiance field [38]. Given a radiance field f and a viewpoint v, R(v, f ) is defined as the image obtained after rendering f from viewpoint v. The rendering loss between the image rendered by radiance fields f and f is formulated as…”

Section: ) Deformed Radiance Field Lossmentioning

confidence: 99%

See 3 more Smart Citations

NeRFlex: Flexible Neural Radiance Fields With Diffeomorphic Deformation

Shin,

Hong,

Lee

2024

IEEE Access

View full text Add to dashboard Cite

Due to the vast array of NeRF-based techniques, the representation power of Neural Radiance Fields (NeRF) has been quickly rising in recent years. However, it is still difficult to offer fresh perspectives for user-controlled geometry alterations with current techniques; instead, they restrict edits to those that were observed during training or to particular regions chosen by the provider. In this paper, we present NeRFlex, Flexible Neural Radiance Fields with Diffeomorphic Deformation, which lets users process geometry completely unrestricted by using radiance field deformation. Given a condition with a specific viewpoint, a conditional score function is estimated using diffusion time steps. The deformation between the radiance fields before and after an edit is then generated by the score function and the initial radiance field. To guarantee topology preservation, invertibility, and smooth transformation, diffeomorphic constraints are provided to the deformation field. Experiments on various objects demonstrate NeRFlex's ability to generate flexible deformations and high-quality novel views after geometry edits, which were never observed in the training data. Continuous deformation along the pathway leading to the deformed object is also obtained by diffeomorphism decomposition.INDEX TERMS NeRF, neural radiance field, diffeomorphic deformation.

show abstract

Section: Diffusion Modelsmentioning

confidence: 99%

Section: Diffusion Modelsmentioning

confidence: 99%

Section: ) Radiance Fieldmentioning

confidence: 99%

Section: ) Deformed Radiance Field Lossmentioning

confidence: 99%

See 2 more Smart Citations

NeRFlex: Flexible Neural Radiance Fields With Diffeomorphic Deformation

Shin,

Hong,

Lee

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Some modifications of NeRFs [16,42,6] and NeRF-based generative models [8,21] allows for the explicit expression and appearance control of the rendered faces. Further, recently demonstrated abilities of diffusion models to generate highly accurate 2D images are currently being transferred onto 3D objects [59,36] and 3D human heads [54]. GAN Inversion.…”

Section: Related Workmentioning

confidence: 99%

TriPlaneNet: An Encoder for EG3D Inversion

Bhattarai¹,

Nießner²,

Sevastopolsky³

2023

Preprint

View full text Add to dashboard Cite

Recent progress in NeRF-based GANs has introduced a number of approaches for high-resolution and high-fidelity generative modeling of human heads with a possibility for novel view rendering. At the same time, one must solve an inverse problem to be able to re-render or modify an existing image or video. Despite the success of universal optimization-based methods for 2D GAN inversion, those, applied to 3D GANs, may fail to produce 3D-consistent renderings. Fast encoder-based techniques, such as those developed for StyleGAN, may also be less appealing due to the lack of identity preservation. In our work, we introduce a real-time method that bridges the gap between the two approaches by directly utilizing the tri-plane representation introduced for EG3D generative model. In particular, we build upon a feed-forward convolutional encoder for the latent code and extend it with a fully-convolutional predictor of tri-plane numerical offsets. As shown in our work, the renderings are similar in quality to optimizationbased techniques and significantly outperform the baselines for novel view. As we empirically prove, this is a consequence of directly operating in the tri-plane space, not in the GAN parameter space, while making use of an encoderbased trainable approach.

show abstract

DiffRF: Rendering-Guided 3D Radiance Field Diffusion

Cited by 2 publications

References 0 publications

NeRFlex: Flexible Neural Radiance Fields With Diffeomorphic Deformation

NeRFlex: Flexible Neural Radiance Fields With Diffeomorphic Deformation

TriPlaneNet: An Encoder for EG3D Inversion

Contact Info

Product

Resources

About