Crowdsampling the Plenoptic Function

Li, Zhengqi; Xian, Wenqi; Davis, Abe; Snavely, Noah

doi:10.1007/978-3-030-58452-8_11

Cited by 60 publications

(50 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The goal is to infer the scene structure and view-dependent appearance given a set of input views. Prior work reasons over an explicit [11,15,22,72] or discrete volumetric [20,33,36,40,44,55,64,65,67,68,80,82] representation of the underlying geometry. However, both have fundamental limitations -explicit representations often require fixing the structure's topology and have poor local optima, while discrete volumetric approaches scale poorly to higher resolutions.…”

Section: Related Workmentioning

confidence: 99%

Editing Conditional Radiance Fields

Liu

Zhang

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

116

View full text Add to dashboard Cite

A neural radiance field (NeRF) is a scene model supporting high-quality view synthesis, optimized per scene. In this thesis, we explore enabling user editing of a category-level NeRF -also known as a conditional radiance field -trained on a shape category. Specifically, we introduce a method for propagating coarse 2D user scribbles to the 3D space, to modify the color or shape of a local region. First, we propose a conditional radiance field that incorporates new modular network components, including a shape branch that is shared across object instances. Observing multiple instances of the same category, our model learns underlying part semantics without any supervision, thereby allowing the propagation of coarse 2D user scribbles to the entire 3D region (e.g., chair seat). Next, we propose a hybrid network update strategy that targets specific network components, which balances efficiency and accuracy. During user interaction, we formulate an optimization problem that both satisfies the user's constraints and preserves the original object structure. We demonstrate our approach on various editing tasks over three shape datasets and show that it outperforms prior neural editing approaches. Finally, we edit the appearance and shape of a real photograph and show that the edit propagates to extrapolated novel views.

show abstract

Section: Related Workmentioning

confidence: 99%

Editing Conditional Radiance Fields

Liu

Zhang

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

116

View full text Add to dashboard Cite

show abstract

“…Recent works [31,32] on view synthesis often require retraining for any testing scene. Although, they produce visually impressive novel views but they are not able to generalize to unseen data.…”

Section: Comparing With State-of-the-art Methodsmentioning

confidence: 99%

“…Recent neural rendering methods have introduced a generative model that understands the underlying 3D scene structure and faithfully produces the target view at the distant query pose [31,32,33,34]. Generative Query Network (GQN) [6] and its variant [7,8,9] are incorporating all input observation (images and poses) into a single implicit 3D scene representation to generate the target view.…”

Section: Related Workmentioning

confidence: 99%

Sequential View Synthesis with Transformer

Nguyen-Ha

Huynh

Rahtu

et al. 2021

Computer Vision – ACCV 2020

View full text Add to dashboard Cite

This paper addresses the problem of novel view synthesis by means of neural rendering, where we are interested in predicting the novel view at an arbitrary camera pose based on a given set of input images from other viewpoints. Using the known query pose and input poses, we create an ordered set of observations that leads to the target view. Thus, the problem of single novel view synthesis is reformulated as a sequential view prediction task. In this paper, the proposed Transformer-based Generative Query Network (T-GQN) extends the neural-rendering methods by adding two new concepts. First, we use multi-view attention learning between context images to obtain multiple implicit scene representations. Second, we introduce a sequential rendering decoder to predict an image sequence, including the target view, based on the learned representations. Finally, we evaluate our model on various challenging datasets and demonstrate that our model not only gives consistent predictions but also doesn't require any retraining for finetuning.

show abstract

“…Although image-based rendering methods have a big range of renderable viewpoints, they tend to be sensitive to the quality of precomputed depth maps. Recently, instead of predicting proxy geometry individually, some methods [19,21,22,39,41] propose to optimize 3D representations jointly with differentiable renderers from input images. An emerging direction is using MLPs as implicit representations, where the MLP networks map continuous spatial points to target values, including signed distance [30], occupancy [26], and radiance [28].…”

Section: Related Workmentioning

confidence: 99%

Efficient Neural Radiance Fields with Learned Depth-Guided Sampling

Lin¹,

Peng²,

Xu³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper aims to reduce the rendering time of generalizable radiance fields. Some recent works equip neural radiance fields with image encoders and are able to generalize across scenes, which avoids the per-scene optimization. However, their rendering process is generally very slow. A major factor is that they sample lots of points in empty space when inferring radiance fields. In this paper, we present a hybrid scene representation which combines the best of implicit radiance fields and explicit depth maps for efficient rendering. Specifically, we first build the cascade cost volume to efficiently predict the coarse geometry of the scene. The coarse geometry allows us to sample few points near the scene surface and significantly improves the rendering speed. This process is fully differentiable, enabling us to jointly learn the depth prediction and radiance field networks from only RGB images. Experiments show that the proposed approach exhibits state-of-the-art performance on the DTU, Real Forward-facing and NeRF Synthetic datasets, while being at least 50 times faster than previous generalizable radiance field methods. We also demonstrate the capability of our method to synthesize free-viewpoint videos of dynamic human performers in real-time. The code will be available at https://zju3dv.github.io/enerf/.

show abstract

Crowdsampling the Plenoptic Function

Cited by 60 publications

References 64 publications

Editing Conditional Radiance Fields

Editing Conditional Radiance Fields

Sequential View Synthesis with Transformer

Efficient Neural Radiance Fields with Learned Depth-Guided Sampling

Contact Info

Product

Resources

About