EPI-based Oriented Relation Networks for Light Field Depth Estimation

Abstract

Light field cameras record not only the spatial information of observed scenes but also the directions of all incoming light rays. The spatial and angular information implicitly contain geometrical characteristics such as multi-view or epipolar geometry, which can be exploited to improve the performance of depth estimation. An Epipolar Plane Image (EPI), the unique 2D spatial-angular slice of the light field, contains patterns of oriented lines. The slope of these lines is associated with the disparity. Benefiting from this property of EPIs, some representative methods estimate depth maps by analyzing the disparity of each line in EPIs. However, these methods often extract the optimal slope of the lines from EPIs while ignoring the relationship between neighboring pixels, which leads to inaccurate depth map predictions. Based on the observation that an oriented line and its neighboring pixels in an EPI share a similar linear structure, we propose an end-to-end fully convolutional network (FCN) to estimate the depth value of the intersection point on the horizontal and vertical EPIs. Specifically, we present a new feature-extraction module, called Oriented Relation Module (ORM), that constructs the relationship between the line orientations. To facilitate training, we also propose a refocusing-based data augmentation method to obtain different slopes from EPIs of the same scene point. Extensive experiments verify the efficacy of learning relations and show that our approach is competitive to other state-of-the-art methods. The code and the trained models are available at https://github.com/lkyahpu/EPI_ORM.git.

Depth Estimation for Integral Imaging Microscopy Using a 3D–2D CNN with a Weighted Median Filter

“…We compared RMSE values with the state-of-the-art algorithms. In these cases, our method is compared with EPI-ORM [ 32 ], EPINET [ 27 ], MANET [ 38 ], and 3D-CNN-LF Depth [ 39 ]. The results of the qualitative evaluation (boxes, dino, dots, pyramids, and town) are shown in Table 1 .…”

“…To evaluate our method for estimating light-field depth, we compared it with state-of-the-art algorithms. In these cases, EPI-ORM [ 32 ], EPINET [ 27 ], MANET [ 38 ], and 3D-CNN-LF Depth [ 39 ] were quantified with a discrete entropy (DE) test. All specimens are shown in Figure 7 .…”

“…Convolutional Neural Networks (CNNs) have achieved state-of-the-art performances for segmentation [ 25 ], super-resolution [ 26 ], depth estimation [ 27 , 28 , 29 , 30 ], and 3D reconstruction [ 31 ] using EPI-based methods. Li et al [ 32 ] introduced an end-to-end fully convolutional network (FCN) for estimating the depth value as nearby pixels in an EPI have a similar linear structure called oriented relation module (ORM). However, these EPI-based approaches only take the EPI features of horizontal and vertical directions, which is insufficient information and decreases the accuracy of the results.…”

“…( a ) Central directional view image. Depth images generated by ( b ) EPI-ORM [ 32 ], ( c ) EPINET [ 27 ], ( d ) MANET [ 38 ], ( e ) 3D-CNN-LF Depth [ 39 ], ( f ) proposed methods, from the same input images.…”

“… Discrete entropy values for gear, microchip, and seedpod using EPI-ORM [ 32 ], EPINET [ 27 ], MANET [ 38 ], and 3D-CNN-LF Depth [ 39 ] proposed depth estimation methods. …”