Unsupervised Dense Light Field Reconstruction with Occlusion Awareness

Abstract

Light field (LF) reconstruction is a fundamental technique in light field imaging and has applications in both software and hardware aspects. This paper presents an unsupervised learning method for LF‐oriented view synthesis, which provides a simple solution for generating quality light fields from a sparse set of views. The method is built on disparity estimation and image warping. Specifically, we first use per‐view disparity as a geometry proxy to warp input views to novel views. Then we compensate the occlusion with a network by a forward‐backward warping process. Cycle‐consistency between different views are explored to enable unsupervised learning and accurate synthesis. The method overcomes the drawbacks of fully supervised learning methods that require large labeled training dataset and epipolar plane image based interpolation methods that do not make full use of geometry consistency in LFs. Experimental results demonstrate that the proposed method can generate high quality views for LF, which outperforms unsupervised approaches and is comparable to fully‐supervised approaches.

Real-time per-pixel focusing method for light field rendering

“…dataset, using the strongest competitor, the shearlet approach [22], takes 5 s on a GeForce GTX Titan X, which is unsuitable for real-time rendering. The proposed method does not reach the same visual quality as newer learning based methods [28] when measured on the same dataset that was used in the original paper, but slightly outperforms older methods [39] (indirect comparison on Kitchen and Museum datasets, difference about 1 dB [28]). The proposed method, however, does not depend on any training process.…”

“…Deep learning approaches were also utilized to address the depth extraction [24,25] and rendering [26,27], based on a few reference images. An unsupervised approach working with planar light fields, using one network for disparity and one for occlusion map estimation, managed to yield results comparable to supervised approaches, overcoming the fully supervised methods' drawbacks [28].…”

“…Visual comparison with other state of the art methods: Vagharshakyan [22], Shi [21], Brox [9], and Ni [28], rendering a new synthetic view. Our method outperforms other general methods but does not reach the same quality as learning based methods, trained on the specific dataset.…”

A Learned Compact and Editable Light Field Representation

“…[20] employ spatial-angular alternating convolutions within a residual learning framework. Different from these methods requiring supervision of dense light fields, Ni et al [21] utilize inter-view cycle consistency to enable the unsupervised learning of a light field from two input views, where occlusions are compensated through a forwardbackward warping scheme. Reducing the amount of input views to one, Srinivasan et al [22] explores the task of synthesizing a light field from a single image, achieving impressive results for very specific scenes.…”