Learning Fused Pixel and Feature-Based View Reconstructions for Light Fields

Abstract

In this paper, we present a learning-based framework for light field view synthesis from a subset of input views. Building upon a lightweight optical flow estimation network to obtain depth maps, our method employs two reconstruction modules in pixel and feature domains respectively. For the pixel-wise reconstruction, occlusions are explicitly handled by a disparity-dependent interpolation filter, whereas inpainting on disoccluded areas is learned by convolutional layers. Due to disparity inconsistencies, the pixel-based reconstruction may lead to blurriness in highly textured areas as well as on object contours. On the contrary, the featurebased reconstruction well performs on high frequencies, making the reconstruction in the two domains complementary. End-to-end learning is finally performed including a fusion module merging pixel and feature-based reconstructions. Experimental results show that our method achieves state-of-the-art performance on both synthetic and realworld datasets, moreover, it is even able to extend light fields' baseline by extrapolating high quality views without additional training.

Fast and Accurate Light Field View Synthesis by Optimizing Input View Selection

“…Other depth estimation based light field angular super-resolution reconstruction methods [ 18 , 23 ] also show that different input views have an impact on the reconstruction of novel views, and in this paper, we undertake a specific investigation.…”

“…Many scholars have studied how to improve the angular resolution while ensuring the high spatial resolution, which is also called light field view synthesis [ 16 , 17 , 18 , 19 ]. Wanner et al [ 20 ] first estimate the depth at the input views and use it to warp the input images to the novel view.…”

“…Wu et al [ 22 ] model the problem as a learning-based angular detail restoration on EPI. Nevertheless, the method relying on EPI structures works well only if the baseline is small [ 18 ].…”

“…Li Kun et al [ 24 ] synthesized high-quality novel views by CNN with Resnet blocks. Shi et al [ 18 ] used a fusion module to merge pixel and feature-based view synthesis for light fields, improving the quality of synthesized novel views. They also designed an interpolation method to solve the occlusion problem in view synthesis.…”

“…For instance, in a 3D display, enough views and spatial resolution are essential to provide a truly immersive 3D experience [15]. Many scholars have studied how to improve the angular resolution while ensuring the high spatial resolution, which is also called light field view synthesis [16][17][18][19]. Wanner et al [20] first estimate the depth at the input views and use it to warp the input images to the novel view.…”