4D Temporally Coherent Light-Field Video

Abstract

Light-field video has recently been used in virtual and augmented reality applications to increase realism and immersion. However, existing light-field methods are generally limited to static scenes due to the requirement to acquire a dense scene representation. The large amount of data and the absence of methods to infer temporal coherence pose major challenges in storage, compression and editing compared to conventional video. In this paper, we propose the first method to extract a spatio-temporally coherent light-field video representation. A novel method to obtain Epipolar Plane Images (EPIs) from a spare lightfield camera array is proposed. EPIs are used to constrain scene flow estimation to obtain 4D temporally coherent representations of dynamic light-fields. Temporal coherence is achieved on a variety of light-field datasets. Evaluation of the proposed light-field scene flow against existing multiview dense correspondence approaches demonstrates a significant improvement in accuracy of temporal coherence.

Scene Flow Estimation From Sparse Light Fields Using a Local 4D Affine Model

“…The local one is derived from Lucas-Kanade [36] and the global one from Horn-Schunck [37]. The authors in [38] estimate a scene flow to construct a 4D spatiotemporally coherent representation of dynamic scenes from sparse light fields. First, a 3D point cloud is estimated, then every point is back-projected to a more densely sampled virtual light field, and the resulting EPIs are used to compute the scene flow using the oriented window approach [15].…”

Computational Photography : High Dynamic Rangeand Light Fields

“…Capturing high resolution light field and light field videos requires an excessively large amount of bandwidth, which has been the main barrier for light field imaging. Light fields and light field videos have varieties of applications such as refocusing [21], interactive 3D video [223], free-viewpoint television (FTV) [224], light field display [225], auto-stereoscopic 3D display [226], depth estimation and occlusion modeling [227,228,229], scene flow estimation [230], light field segmentation [231], stitching [232], material recognition [233], compositing light field video [217], and photo-real digital actors [234].…”

A Self-regulating Spatio-Temporal Filter for Volumetric Video Point Clouds

“…The accuracy of their results is not comparable with the methods mentioned above, but they show good performance with a reduced number of viewpoints and wide baseline. Mustafa et al extended their work to include sequences that are not only temporally but also semantically coherent [26], and even light-field video [28].…”