IBRNet: Learning Multi-View Image-Based Rendering

Abstract

We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. The core of our method is a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations (3D spatial locations and 2D viewing directions), drawing appearance information on the fly from multiple source views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering (IBR), and allows us to render high-resolution imagery. Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes. We render images using classic volume rendering, which is fully differentiable and allows us to train using only multiview posed images as supervision. Experiments show that our method outperforms recent novel view synthesis methods that also seek to generalize to novel scenes. Further, if fine-tuned on each scene, our method is competitive with state-of-the-art single-scene neural rendering methods. 1

Fast Training of Neural Lumigraph Representations using Meta Learning

“…We demonstrate this by comparison to several state-of-the-art methods. Specifically, we evaluate the volumetric representation of NeRF [59], a meshbased representation similar to SVS [8], the neural signed distance function-based representations of IDR [4] and NLR [5], and the image-based rendering of IBRNet [28]. For SVS [8] we use our own simplified implementation and denote it SVS*.…”

“…Recent IBR techniques leverage neural networks to learn the required blending weights [19][20][21][22][23][24]. These neural IBR methods either use proxy geometry, for example obtained by SfM or MVS [25,26], together with on-surface feature aggregation [7,8] or use learned pixel aggregation functions [27,28] for geometry-free image-based view synthesis. Our approach is closely related to the geometry-assisted and feature-interpolating view synthesis techniques.…”

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

“…Baselines. Among the recent generalizable NeRF methods [36,52,47], we compare with Pixel-NeRF [52] and PVA [36] which focus on very sparse (up to 3 or 4) input views. we reimplement [36] since it is not open-sourced.…”

“…Despite the promising results, these general NeRF [19,53] and human-specific NeRF [13,32,33,35,50] methods must be optimized for each new video separately, and generalize poorly on unseen scenarios. Generalizable NeRFs [36,47,52] try to avoid the expensive per-scene optimization by imageconditioning using pixel-aligned features. However, directly extending such methods to model complex and dynamic 3D humans is not straightforward when available observations are highly sparse.…”

“…Recently, neural radiance fields (NeRF) [28,13,19,32,33,35,36,47,50,52,53] have shown photo-realistic novel view synthesis results in per-scene optimization settings. To avoid the expensive per-scene training and improve the practicality, generalizable NeRFs [36,52,47] have been proposed which use image-conditioned, pixel-aligned features and achieve feed-forward view synthesis from sparse input views [36,52]. Direct application of these methods to complex and non-rigid human motion is not straightforward, however, and naive solutions suffer from significant artifacts as shown in Fig.…”