Deep blending for free-viewpoint image-based rendering

Abstract

Free-viewpoint image-based rendering (IBR) is a standing challenge. IBR methods combine warped versions of input photos to synthesize a novel view. The image quality of this combination is directly affected by geometric inaccuracies of multi-view stereo (MVS) reconstruction and by view- and image-dependent effects that produce artifacts when contributions from different input views are blended. We present a new deep learning approach to blending for IBR, in which we use held-out real image data to learn blending weights to combine input photo contributions. Our Deep Blending method requires us to address several challenges to achieve our goal of interactive free-viewpoint IBR navigation. We first need to provide sufficiently accurate geometry so the Convolutional Neural Network (CNN) can succeed in finding correct blending weights. We do this by combining two different MVS reconstructions with complementary accuracy vs. completeness tradeoffs. To tightly integrate learning in an interactive IBR system, we need to adapt our rendering algorithm to produce a fixed number of input layers that can then be blended by the CNN. We generate training data with a variety of captured scenes, using each input photo as ground truth in a held-out approach. We also design the network architecture and the training loss to provide high quality novel view synthesis, while reducing temporal flickering artifacts. Our results demonstrate free-viewpoint IBR in a wide variety of scenes, clearly surpassing previous methods in visual quality, especially when moving far from the input cameras.

Pushing the Boundaries of View Extrapolation With Multiplane Images

“…Many successful modern approaches to view synthesis [3,13,21,40] follow a strategy of computing detailed local geometry for each input view followed by forward projecting and blending the local texture from multiple input views to render a novel viewpoint. This research has traditionally focused on interpolating between input views and therefore does not attempt to predict content that is occluded in all input images.…”

DeepVoxels: Learning Persistent 3D Feature Embeddings

“…Some approaches rely on light fields [24]. Recently, deep-learning has been used to aid image-based rendering via learning a small subtask, i.e., the computation of the blending weights [16,11]. While this can achieve photorealism, it depends on a dense set of high-resolution photographs to be available at rendering time and requires an error prone reconstruction step to obtain the geometric proxy.…”

VR content creation and exploration with deep learning: A survey

“…The method is aware of occlusion and improves results on challenging scenes with thin structures and high depth complexity. Hedman et al [61] proposed DeepBlending, a CNN-based IBR blending system with per-view geometry refinement and geometry-aware mesh simplification for quality improvement. Given a set of photos from several views, the proposed blending network takes selected warped view mosaics and a global mesh rendering, and outputs weights for blending each pixel from the views (see Fig.…”