Deep Light Field Acquisition Using Learned Coded Mask Distributions for Color Filter Array Sensors

Abstract

Compressive light field photography enables light field acquisition using a single sensor by utilizing a color coded mask. This approach is very cost effective since consumer-level digital cameras can be turned into light field cameras by simply placing a coded mask between the sensor and the aperture plane. This paper describes a deep learning architecture for compressive light field acquisition using a color coded mask and a sensor with Color Filter Array (CFA). Unlike previous methods where a fixed mask pattern is used, our deep network learns the optimal distribution of the color coded mask pixels. The proposed solution enables end-to-end learning of the color-coded mask distribution and the reconstruction network, taking into account the sensor CFA. Consequently, the resulting network can efficiently perform joint demosaicing and light field reconstruction of images acquired with color-coded mask and a CFA sensor. Compared to previous methods based on deep learning with monochrome sensors, as well as traditional compressive sensing approaches using CFA sensors, we obtain superior color reconstruction of the light fields.

Deep Unrolling for Light Field Compressed Acquisition Using Coded Masks

“…This architecture is based on an unrolled HQS optimization method with a learned proximal operator. • In the 1-shot case, we show that our architecture yields a 2.5 dB improvement over the deep learning based method in [16], which is already in average 1.97 dB better than the method in [15], using the same acquisition scheme. We also show that our approach yields a significant improvement in PSNR compared to traditional iterative methods using the same acquisition scheme.…”

“…Whereas we could use fixed values for the pixels of the color-coded mask, for the sake of comparison, we instead follow the approach of [15] and [16], in which a different mask is generated each time, by independently drawing the transmittance value t ∈ R Λ of each of its pixels from a distribution D. Besides, to be able to compare with [15] and [16], we depart from physical realizability in our experiments (unless stated otherwise) by directly drawing a value t ∼ D independently for each element of the modulation field (i.e. for each spatial-angular coordinate (x, u)), instead of computing ψ using Equation ( 6).…”

“…While traditional methods using sparse priors have been to some extent successfully applied to light field reconstruction [11]- [14], the iterative nature of the reconstruction algorithm greatly precludes its actual use, notably in real-time applications. Alternatively, deep learning methods have been very successful at reconstructing light fields in a way that is both fast and accurate [15], [16]. These methods are orders of magnitude faster than traditional iterative methods like the orthogonal matching pursuit [17], and can outperform sparsity priors for a great number of VOLUME 4, 2016 image reconstruction tasks [18].…”

“…We show that our approach can be applied to any number of measurements and give experimental results for a number of measurements going from 1 to 3. Whereas the methods in [15], [16] are based on convolutional neural networks (CNN) that act as regression networks, we derive instead our architecture from the unrolling of the minimization of a regularized mean-squared problem. The proposed solution is actually based on an unrolled Half Quadratic Splitting (HQS) optimization method with a learned proximal operator, leading to a completely different architecture.…”

“…using a coded mask placed on the sensor in comparison with coded aperture designs. For the sake of comparison with the methods in [14], [15] and [16] which also consider masks placed on the sensor, we first consider a mask drawn from a uniform distribution, with a model assuming the presence of pinholes on the aperture plane. This corresponds to the case where the real continuous light field is first discretized and then modulated with the coded mask and the color filter array.…”