A Unified Learning-Based Framework for Light Field Reconstruction From Coded Projections

Abstract

Light fields present a rich way to represent the 3D world by capturing the spatio-angular dimensions of the visual signal. However, the popular way of capturing light fields (LF) via a plenoptic camera presents a spatio-angular resolution tradeoff. Computational imaging techniques such as compressive light field and programmable coded aperture reconstruct full sensor resolution LF from coded projections obtained by multiplexing the incoming spatio-angular light field. Here, we present a unified learning framework that can reconstruct LF from a variety of multiplexing schemes with minimal number of coded images as input. We consider three light field capture schemes: heterodyne capture scheme with code placed near the sensor, coded aperture scheme with code at the camera aperture and finally the dual exposure scheme of capturing a focus-defocus pair where there is no explicit coding. Our algorithm consists of three stages 1) we recover the all-in-focus image from the coded image 2) we estimate the disparity maps for all the LF views from the coded image and the all-in-focus image, 3) we then render the LF by warping the all-in-focus image using disparity maps. We show that our proposed learning algorithm performs either on par with or better than the state-of-the-art methods for all the three multiplexing schemes. LF from focus-defocus pair is especially attractive as it requires no hardware modification and produces LF reconstructions that are comparable to the current state of the art learning-based view synthesis approaches from multiple images. Thus, our work paves the way for capturing full-resolution LF using conventional cameras such as DSLRs and smartphones.Index Terms-Light field resolution trade-off, compressive light field imaging, coded aperture photography, disparity based view synthesis.

Factorized Modulation For Singleshot Lightfield Acquisition

“…A popular choice is coded aperture imaging [13,14,15,16] (denoted as CA), whose modulation function is given as…”

“…Another choice is codedmask [17,18,15] (denoted as CM), where a lightattenuating mask, p(x, y), is inserted be tween the aperture and imaging planes, allowing more free dom for modulation. Ideally, the modulation function is de fined as…”

“…On the righthand side, the modulation is conducted individually on the aperture and imaging planes in a factorized manner. Accordingly, this modulation can be implemented by combining the architectures for coded aper ture (CA) imaging [13,14,15,16] and pixelwise coded ex posure (CE) imaging [22,23], as shown in Fig. 1.…”

“…Several researchers have utilized a direct approach using multiple cameras [11,12] or a camera mounted on a moving gantry, which is costly in terms of the hardware or the time required to capture an entire LF. To make the acquisition process more efficient, lens arraybased cam eras [5], coded aperture cameras [13,14,15,16], and coded mask cameras [17,18,15] have been investigated. Lens array based cameras enable singleshot acquisition of an entire LF at the cost of the spatial resolution for each viewpoint; in prin ciple, U × V views are obtained in 1/U × 1/V spatial reso lution of the image sensor.…”

“…To achieve this, we propose a new modulation scheme called factorized modulation that can approximate ar bitrary 4D modulation patterns in a factorized manner. The factorized modulation can be hardwareimplemented by com bining architectures for coded aperture imaging [13,14,15,16] and pixelwise coded exposure imaging [22,23]. Follow ing the concept of a previous work [14], the modulation pat tern is jointly optimized with a CNNbased LF reconstruction algorithm.…”