Predicting Sharp and Accurate Occlusion Boundaries in Monocular Depth Estimation Using Displacement Fields

Abstract

Current methods for depth map prediction from monocular images tend to predict smooth, poorly localized contours for the occlusion boundaries in the input image. This is unfortunate as occlusion boundaries are important cues to recognize objects, and as we show, may lead to a way to discover new objects from scene reconstruction. To improve predicted depth maps, recent methods rely on various forms of filtering or predict an additive residual depth map to refine a first estimate. We instead learn to predict, given a depth map predicted by some reconstruction method, a 2D displacement field able to re-sample pixels around the occlusion boundaries into sharper reconstructions. Our method can be applied to the output of any depth estimation method and is fully differentiable, enabling endto-end training. For evaluation, we manually annotated the occlusion boundaries in all the images in the test split of popular NYUv2-Depth dataset. We show that our approach improves the localization of occlusion boundaries for all state-of-the-art monocular depth estimation methods that we could evaluate ([32, 10, 6, 28]), without degrading the depth accuracy for the rest of the images.

Pixel-Pair Occlusion Relationship Map (P2ORM): Formulation, Inference and Application

“…We keep the original training and testing data of BSDS. NYUv2-OR is tested on a subset of NYUv2 [33] with occlusion boundaries from [39] and 1 As DOOBNet and OFNet are coded in Caffe, in order to have an unified platform for experimenting them on new datasets, we carefully re-implemented them in PyTorch (following the Caffe code). We could not reproduce exactly the same quantitative values provided in the original papers (ODS and OIS metrics are a bit less while AP is a bit better), probably due to some intrinsic differences between frameworks Caffe and PyTorch, however, the difference is very small (less than 0.03, cf.…”

“…Depth map refinement. To assess our refinement approach, we compare with [39], which is the current state-of-the-art for depth refinement on boundaries.…”

“…7. Gain in edge quality after depth refinement for metrics acc (left) and comp (right) on NYUv2 (top) for respectively [8,22,40,9,20,58] and on iBimis-1 (bottom) for [8,22,40,27,28,25]: metric on input depth maps (blue), after refining with [39] (orange), and after our refinement (green). Lower metric value is better.…”

“…These methods now underperform state-of-the-art deep-learning-based methods without refinement [20,58] while being more computationally expensive. Recently, [39] predicts image displacement fields to sharpen initial depth predictions.…”

The Future of Endoscopic Navigation: A Review of Advanced Endoscopic Vision Technology

“…Since an accurate depth value is difficult to obtain in large-scale training, an unsupervised frame was developed that could use spatial (between left-right pairs) or temporal (forwardbackward pairs) photometric warp error [114][115][116] or both [117]. The latest research [118] sought to obtain clear boundaries in estimating depth by resampling pixels around occlusion boundaries. One obstacle in training is the patientspecific nature of the texture of the tissue when first used in depth reconstruction in colonoscopy [119].…”