Simultaneous estimation of super-resolved depth and all-in-focus images from a plenoptic camera

Abstract

This paper presents a new technique to simultaneously estimate the depth map and the all-in-focus image of a scene, both at super-resolution, from a plenoptic camera.A plenoptic camera uses a microlens array to measure the radiance and direction of all the light rays in a scene. It is composed of n×n microlenses and each of them generates a m×m image. Previous approaches to the depth and all-infocus estimation problem processed the plenoptic image, generated a n×n×m focal stack, and were able to obtain a n×n depth map and all-in-focus image of the scene. This is a major drawback of the plenoptic camera approach to 3DTV since the total resolution of the camera n 2 m 2 is divided by m 2 to obtain a final resolution of n 2 pixels. In our approach we propose a new super-resolution focal stack that is combined with multiview depth estimation. This technique allows a theoretical resolution of approximately n 2 m 2 /4 pixels. This is an o(m 2 ) increment over previous approaches.From a practical point of view, in typical scenes we are able to increase 25 times the resolution of previous techniques. The time complexity of the algorithm makes possible to obtain real-time processing for 3DTV using appropriate hardware (GPU's or FPGA's) so it could be used in plenoptic video-cameras.

Geometric Calibration of Micro-Lens-Based Light-Field Cameras Using Line Features

“…A major drawback of such hand-held systems is low spatial resolution. In order to overcome this limitation, several light-field superresolution methods have been developed [3,11]. Light-field panorama [2] and disparity estimation [13,16] are also interesting applications.…”

A Light Transport Framework for Lenslet Light Field Cameras

“…This is observed in the basic lensletbased design [Perez Nava and Luke 2009], the focused light field camera [Georgiev et al 2011 …”

“…The projection algorithm recently gained popularity owing to its simplicity and efficiency [Kitamura et al 2004;Perez Nava and Luke 2009;Georgiev et al 2011;Yu et al 2012]. In contrast, other works rely on expensive deconvolution computations, and usually require sophisticated image priors to regularize the process Bishop and Favaro 2012;Wanner and Goldluecke 2012b;Marwah et al 2013].…”

“…It is known that in practice, this projection algorithm can create images with resolution higher than the lenslet resolution [Perez Nava and Luke 2009;Georgiev et al 2011]. Intuitively, if the scene is Lambertian, a sample at (x, u) can represent all samples with identical x coordinates.…”

“…Since the light field must be further processed for display, we extend our framework to analyze two main categories of rendering algorithms: the simple projection-based algorithm [Kitamura et al 2004;Chan et al 2007;Perez Nava and Luke 2009;Georgiev et al 2011] and the inverse light transport (deconvolution) process [Bishop and Favaro 2012;Shroff and Berkner 2013;Broxton et al 2013]. For the projection-based algorithm, we derive the overall filtering kernel for the rendered image and show that it can generate high-resolution results for most designs.…”