EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images

Abstract

Light field cameras capture both the spatial and the angular properties of light rays in space. Due to its property, one can compute the depth from light fields in uncontrolled lighting environments, which is a big advantage over active sensing devices. Depth computed from light fields can be used for many applications including 3D modelling and refocusing. However, light field images from hand-held cameras have very narrow baselines with noise, making the depth estimation difficult. Many approaches have been proposed to overcome these limitations for the light field depth estimation, but there is a clear trade-off between the accuracy and the speed in these methods. In this paper, we introduce a fast and accurate light field depth estimation method based on a fully-convolutional neural network. Our network is designed by considering the light field geometry and we also overcome the lack of training data by proposing light field specific data augmentation methods. We achieved the top rank in the HCI 4D Light Field Benchmark on most metrics, and we also demonstrate the effectiveness of the proposed method on real-world light-field images.

Learning to Think Outside the Box: Wide-Baseline Light Field Depth Estimation with EPI-Shift

“…For the quantitative evaluation in Figure 7, we plot 13 error measures from the HCI 4D Light Field Benchmark, for details please refer to [10]. Our method outperforms EPINET-Cross [20], in 11 out of 13 metrics with a close tie of the other two. Because our network is run several times (once for each EPI-Shift), the runtime increases linearly with the disparity range and is therefore slightly higher compared to [20].…”

“…For another 30000 iterations, we decreased the learning rate to 10 −5 and fixed the learned batch normalization parameters. We apply a large variety of data augmentation, comparable to [20], including random color channel re-distribution, random brightness and contrast adjustments, random rotations by multiples of 90 • , random scales between 0.5 and 1 and random crops to a patch size of 225 × 225. This patch size leverages the utilization of global information by the U-Net.…”

“…We also evaluated our CNN architecture, without EPI-Shift, similar to [20]. This model only reached an MSE score of 31.15 compared to 0.85 for our model with EPI-Shift.…”

“…Light field depth estimation can be seen as a well-structured multi-view stereo approach. Although the hardware setup of a light field is more involved than a stereo setup, the depth information "encoded" in light fields is considerably more precise and inherently less sensi-(a) Center view of real (left) and synthetic (right) light field (b) EPINET-Cross [20] (c) Our EPI-Shift The current state-of-the-art method EPINET-Cross [20] has only been trained for the small disparity range. It therefore fails at extreme disparities in the synthetic image (right, background) and outside of the trained range in the real image (left, foreground).…”

“…• A network architecture, which enables improved long-range reasoning by combining a feature extraction network [20] with a subsequent U-Net architecture [17][8] for excellent long-range smoothing with low artifacts.…”