Super Resolution of Light Field Images Using Linear Subspace Projection of Patch-Volumes

Abstract

Abstract-Light field imaging has emerged as a very promising technology in the field of computational photography. Cameras are becoming commercially available for capturing real-world light fields. However, capturing high spatial resolution light fields remains technologically challenging, and the images rendered from real light fields have today a significantly lower spatial resolution compared to traditional 2D cameras. This paper describes an example-based super-resolution algorithm for light fields, which allows the increase of the spatial resolution of the different views in a consistent manner across all sub-aperture images of the light field. The algorithm learns linear projections between subspaces of reduced dimension in which reside patch-volumes extracted from the light field. The method is extended to cope with angular super-resolution, where 2D patches of intermediate sub-aperture images are approximated from neighbouring subaperture images using multivariate ridge regression. Experimental results show significant quality improvement when compared to state-of-the-art single-image super-resolution methods applied on each view separately, as well as when compared to a recent light field super-resolution techniques based on deep learning.

A simple framework to leverage state-of-the-art single-image super-resolution methods to restore light fields

“…We compare the performance of our proposed PB-VDSR method against some of the best performing methods in the field of light field super-resolution, namely the CNN based light field super-resolution algorithm (LF-SRCNN) [11], the linear subspace projection based method (BM-PCARR) [9] and the Graph-based light field super resolution (GRAPH) [12]. It must be mentioned that while the BM+PCARR and LF-SRCNN were retrained on 98 light fields that were not considered in the evaluation phase, the network model adopted by VDSR was not retrained on light fields and we used the original model adopted for single image super-resolution.…”

“…Cropped regions of the mean view when using different disparity compensation methods. Underneath each image we provide the average variance across the n angular views which was used in [9] to characterize the performance of the alignment algorithm, where smaller values indicate better alignment.…”

Light Field Super-Resolution using a Low-Rank Prior and Deep Convolutional Neural Networks

“…There are many possible high-resolution light fields I H which can produce the input low-resolution light field I L via the acquisition model defined in (2). Hence, solving this illposed inverse problem requires introducing some priors on I H , which can be a statistical prior such as a GMM model [8], or priors learned from training data as in [5], [10], [11]. Another way to visualize a light field is to consider the EPI representation.…”

“…Instead, we decompose this problem in two sub-problems: i) use optical flow to find the flow matrices u and v that best align each subaperture image with the centre view and ii) use low-rank approximation to derive the rank-k matrix that minimizes the error with respect to the aligned light field. Underneath each image we provide the average variance across the n sub-aperture images which was used in [5] to characterize the performance of the alignment algorithm, where smaller values indicate better alignment.…”

“…However, super-resolving each view separately using state of the art single-image super-resolution techniques would not take advantage of light field properties, in particular of angular redundancy which depends on scene geometry [4]. Moreover, considering each subaperture image as a separate entity may reconstruct light fields which are angularly incoherent [5].…”