Efficient Multi‐image Correspondences for On‐line Light Field Video Processing

Abstract

Figure 1: Our real-time multi-view correspondence algorithm extracts multi-view depth maps from sparse, wide-baseline light field video (here 3×3 cameras), in order to produce high-quality novel views for applications such as virtual apertures or virtual camera positions. AbstractLight field videos express the entire visual information of an animated scene, but their shear size typically makes capture, processing and display an off-line process, i. e., time between initial capture and final display is far from real-time. In this paper we propose a solution for one of the key bottlenecks in such a processing pipeline, which is a reliable depth reconstruction possibly for many views. This is enabled by a novel correspondence algorithm converting the video streams from a sparse array of off-the-shelf cameras into an array of animated depth maps. The algorithm is based on a generalization of the classic multi-resolution Lucas-Kanade correspondence algorithm from a pair of images to an entire array. Special inter-image confidence consolidation allows recovery from unreliable matching in some locations and some views. It can be implemented efficiently in massively parallel hardware, allowing for interactive computations. The resulting depth quality as well as the computation performance compares favorably to other state-of-the art light field-to-depth approaches, as well as stereo matching techniques. Another outcome of this work is a data set of light field videos that are captured with multiple variants of sparse camera arrays.

X-Fields

“…Very similar view positions [Kalantari et al 2016] as for a Lytro camera can be considered dense, while 34 views on a sphere [Lombardi et al 2019] or 40 lights on a hemisphere [Malzbender et al 2001] is sparse. In this paper we focus on wider baselines, with typically × cameras spaced by 5-10 cm [Flynn et al 2019], and respectively a large disparity ranging up to 250 pixels [Dabała et al 2016;Mildenhall et al 2019], where and are single-digit numbers, e.g., 3×3, 5×5 or even 2×1. Depending on resources, a capture setup can be considered simple (cell phone, as we use) or more involved (light stage) as denoted in the "easy capture" column in Tbl.…”

“…Warping and SuperSlowMo first estimate the correspondence in image pairs [Sun et al 2018b] or light field data [Dabała et al 2016] and later apply warping [Mark et al 1997] with ULR-style weights [Buehler et al 2001]. Note how ULR weighting accounts for occlusion.…”

Scene Flow Estimation From Sparse Light Fields Using a Local 4D Affine Model

“…We chose an angular configuration that is similar to the one of real light fields captured by rigs of cameras, such as e.g. in [48] and [49], which respectively provide 5 × 3 and 4 × 4 views. We also use the dataset of [48] to test our method on a real light field sequence: 'Bar'.…”

“…in [48] and [49], which respectively provide 5 × 3 and 4 × 4 views. We also use the dataset of [48] to test our method on a real light field sequence: 'Bar'. Each frame is a 5 × 3 light field, in which each view has a spatial resolution of 1920 × 1080 pixels.…”

“…The optical flows are visualized with the Middlebury color code, and the disparity maps and disparity variations are visualized using a gray-scale representation. The light field frames are taken from[48].…”