Joint Graph-based Depth Refinement and Normal Estimation

Abstract

Depth estimation is an essential component in understanding the 3D geometry of a scene, with numerous applications in urban and indoor settings. These scenes are characterized by a prevalence of human made structures, which in most of the cases, are either inherently piece-wise planar, or can be approximated as such. In these settings, we devise a novel depth refinement framework that aims at recovering the underlying piece-wise planarity of the inverse depth map. We formulate this task as an optimization problem involving a data fidelity term that minimizes the distance to the input inverse depth map, as well as a regularization that enforces a piece-wise planar solution. As for the regularization term, we model the inverse depth map as a weighted graph between pixels. The proposed regularization is designed to estimate a plane automatically at each pixel, without any need for an a priori estimation of the scene planes, and at the same time it encourages similar pixels to be assigned to the same plane. The resulting optimization problem is efficiently solved with ADAM algorithm. Experiments show that our method leads to a significant improvement in depth refinement, both visually and numerically, with respect to state-of-the-art algorithms on Middlebury, KITTI and ETH3D multi-view stereo datasets.

DeepC-MVS: Deep Confidence Prediction for Multi-View Stereo Reconstruction

“…However, the performances of these methods tend to decrease significantly when applied to data even slightly different from that use in the training phase. Therefore, in this article we adopt the methods in [28] and show that, coupled with our proposed confidence, it can improve the 3D reconstruction both qualitatively and quantitatively.…”

“…However, the a priori selection of the possible planes, first, fixes the complexity of the scene in front of the camera and, second, assumes that the selected planes are correct. To overcome the previous limitations, in [28] the authors propose to adopt a regularizer which promotes depth maps fulfilling a piece-wise planar world assumption, but without any a priori candidate plane selection. Also in this method, the confidence plays a fundamental role, as the method im-plicitly fits planes based on the reliable depth areas.…”

“…where u i ∈ R 2 is a vector defining the plane orientation at i. Based on these observations, the authors in [28] propose to refine a noisy and possibly incomplete depth map z by enforcing that the refined inverse depth map d = 1/z is piece-wise planar. The method in [28] requires a depth map confidence: we adopt our proposed one.…”

“…Based on these observations, the authors in [28] propose to refine a noisy and possibly incomplete depth map z by enforcing that the refined inverse depth map d = 1/z is piece-wise planar. The method in [28] requires a depth map confidence: we adopt our proposed one. The refinement of each inverse depth map d = 1/z is cast into the minimization of a cost function comprising two terms.…”

“…This permits to avoid edges between points that are not likely to correspond to the same plane in the scene, as these would just increase the graph size unnecessarily. We refer to [28] for more details on the regularizer in Eqs. ( 5)-( 6) and on the graph construction.…”