Tensor-Based Light Field Compressed Sensing and Epipolar Plane Images Reconstruction via Deep Learning

Abstract

Light field (LF) can capture the spatial and angular information of the light in one single exposure. And the LF images are widely used in various fields, especially in immersive media. The rich imaging information in the LF poses great challenges for transmission. However, LF images are sparse and redundant to some extent, which makes LF compression possible. Besides, the compressed sensing (CS) theory shows that images can be recovered from a small number of measurements when they are sparse. In this paper, we propose a Tensor-based Compressed Sensing method to compress images and Epipolar Plane Images for reconstruction (TCSEPI). This method divides the viewpoints of LF images into several regions and stacks the images in each region into a 4D tensor, which conduct CS together and yields measurements with common characteristics. Subsequently, the epipolar plane images are used to reconstruct the LF images and restore the geometric consistency information. To achieve better reconstruction results, we design two cascaded convolutional neural networks to implement the measurement matrix optimization and LF images reconstruction sequentially. Experimental results show the superior performance of TCSEPI, which achieves at least 3dB gain in PSNR and outperforms state-of-the-art in the reconstruction quality. INDEX TERMS Light field, compressed sensing, epipolar plane images, convolutional neural network, image reconstruction

Learning-based light field imaging: an overview

“…The reconstructions from the two branches are combined to obtain the final spatial-angular super-resolved light field. The use of compressed sensing for light field reconstruction is further expanded using deep learning; In [94], 4D tensors were generated from patches of light field views, and a deep learning scheme embedding 3D convolutions was adopted to build a sparse representation of light fields. Finally, reconstruction is performed by passing the features through a set of fully connected layers and reshaping them into the 4D tensor.…”