DeRF: Decomposed Radiance Fields

Rebain, Daniel; Jiang, Wei; Yazdani, Soroosh; Li, Ké; Yi, Kwang Moo; Tagliasacchi, Andrea

doi:10.1109/cvpr46437.2021.01393

Cited by 130 publications

(87 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A practical challenge in training compositional NeRFs lies in the computational cost of neural volume rendering, as it requires massive queries to render a single pixel. While there have been attempts on fast inference [29,43,37,13,44,62], high space complexity in training remains a challenge. Further, because our perceptual and adversarial losses depend on image patches, the system has to render a large enough patch (instead of a single pixel) at the same time, which further increases its space demand.…”

Section: Model Learningmentioning

confidence: 99%

Unsupervised Discovery of Object Radiance Fields

Yu¹,

Guibas²

2021

Preprint

View full text Add to dashboard Cite

We study the problem of inferring an object-centric scene representation from a single image, aiming to derive a representation that explains the image formation process, captures the scene's 3D nature, and is learned without supervision. Most existing methods on scene decomposition lack one or more of these characteristics, due to the fundamental challenge in integrating the complex 3D-to-2D image formation process into powerful inference schemes like deep networks. In this paper, we propose unsupervised discovery of Object Radiance Fields (uORF), integrating recent progresses in neural 3D scene representations and rendering with deep inference networks for unsupervised 3D scene decomposition. Trained on multi-view RGB images without annotations, uORF learns to decompose complex scenes with diverse, textured background from a single image. We show that uORF performs well on unsupervised 3D scene segmentation, novel view synthesis, and scene editing on three datasets. * * Project website: https://kovenyu.com/uorf/ Preprint. Under review.

show abstract

Section: Model Learningmentioning

confidence: 99%

Unsupervised Discovery of Object Radiance Fields

Yu¹,

Guibas²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Learning 3D scene representation with a parameterized neural network has been largely explored by recent works from various angles such as implicit signed distance function [21,5,34], occupancy [31,35,14], volume rendering (i.e. radiance field) [32,26,33,37,50,44], and shapes [2,15,14]. Such implicit neural representation also started to influence traditional 2D tasks such as image representation [23,40], super-resolution [9], and medical image analysis [49].…”

Section: Rendering With Spatial Encodingmentioning

confidence: 99%

“…Learning how to reconstruct the high-frequency part of the image is the key in the SR task. Many recent works [32,37,50,44] have shown that a carefully designed spatial encoding can help the network recover fine details in 3D scenes. LIIF [9] overlooked the importance of these spatial encoding by directly feeding coordinates into the implicit image function represented by a vanilla MLP.…”

Section: Periodic Spatial Encodingmentioning

confidence: 99%

See 1 more Smart Citation

UltraSR: Spatial Encoding is a Missing Key for Implicit Image Function-based Arbitrary-Scale Super-Resolution

Xu¹,

Wang²

2021

Preprint

View full text Add to dashboard Cite

The recent success of NeRF and other related implicit neural representation methods has opened a new path for continuous image representation, where pixel values no longer need to be looked up from stored discrete 2D arrays but can be inferred from neural network models on a continuous spatial domain. Although the recent work LIIF has demonstrated that such novel approach can achieve good performance on the arbitrary-scale super-resolution task, their upscaled images frequently show structural distortion due to the faulty prediction on high-frequency textures. In this work, we propose UltraSR, a simple yet effective new network design based on implicit image functions in which spatial coordinates and periodic encoding are deeply integrated with the implicit neural representation. We show that spatial encoding is indeed a missing key towards the next-stage high-accuracy implicit image function through extensive experiments and ablation studies. Our UltraSR sets new state-of-the-art performance on the DIV2K benchmark under all super-resolution scales comparing to previous state-of-the-art methods. UltraSR also achieves superior performance on other standard benchmark datasets in which it outperforms prior works in almost all experiments. Our code will be released at https://github.com/SHI-Labs/UltraSR-Arbitrary-Scale-Super-Resolution.

show abstract

“…the recent success in compactly storing 3D field functions within neural networks (e.g. occupancy[Chen and Zhang 2019;, signed distance fields[Atzmon and Lipman 2020;Park et al 2019], and radianceRebain et al 2021]), we propose to store the medial field within the parameters 𝜃 of a deep neural network 𝑀 𝜃 (𝑥). While it is in theoryInput: Ray direction 𝑑 and origin 𝑜 Output: Position 𝑥 of the ray intersection with 𝜕O 𝑥 = 𝑜 repeat 𝑥 ← 𝑥 + Φ(𝑥)𝑑 until |Φ(𝑥)| < 𝜖;…”

mentioning

confidence: 99%