Semantically Derived Geometric Constraints for MVS Reconstruction of Textureless Areas

Abstract

Conventional multi-view stereo (MVS) approaches based on photo-consistency measures are generally robust, yet often fail in calculating valid depth pixel estimates in low textured areas of the scene. In this study, a novel approach is proposed to tackle this challenge by leveraging semantic priors into a PatchMatch-based MVS in order to increase confidence and support depth and normal map estimation. Semantic class labels on image pixels are used to impose class-specific geometric constraints during multiview stereo, optimising the depth estimation on weakly supported, textureless areas, commonly present in urban scenarios of building facades, indoor scenes, or aerial datasets. Detecting dominant shapes, e.g., planes, with RANSAC, an adjusted cost function is introduced that combines and weighs both photometric and semantic scores propagating, thus, more accurate depth estimates. Being adaptive, it fills in apparent information gaps and smoothing local roughness in problematic regions while at the same time preserves important details. Experiments on benchmark and custom datasets demonstrate the effectiveness of the presented approach.

SFA-MDEN: Semantic-Feature-Aided Monocular Depth Estimation Network Using Dual Branches

“…The reason is that Yue et al [50] established a semantic weight map to optimize the estimated depth map, which takes advantage of semantic information as well. Even so, there is a narrow margin between our SFA-MDEN and Yue et al [50] in metrics of Sq Rel, RMSE log, and δ < 1.25 3 . Furthermore, the Abs Rel of SFA-MDEN is 0.023 smaller, the RMSE of SFA-MDEN is 0.135 m smaller, and the threshold accuracy δ < 1.25 is 2.9% higher compared to Yue et al [50].…”

“…Moreover, our model ranks second in Sq Rel and RMSE. Although SFA-MDEN comes last in RMSE log and δ < 1.25 3 , there is a narrow margin between these methods. In conclusion, our method has achieved a competitive performance compared to the state-of-the-art methods with semantic auxiliary.…”

“…Threshold accuracy (%correct): δ = max( D î D i ,D i D i ) < T (T is the threshold and can be assigned as 1.25, 1.25 2 , 1.25 3 ) whereD i denotes the predicted depth, D i denotes the ground truth depth, the subscripts i and j are the pixel index, and N represents the total number of valid pixels.…”

“…Depth estimation plays a fundamental role in numerous application scenarios, such as image reconstruction [ 1 ], object detection [ 2 ], semantic segmentation [ 3 ], pose estimation [ 4 ], and medical image processing [ 5 ]. There are various sensors available to obtain depth information based on either active or passive measurement methods.…”

DP-MVS: Detail Preserving Multi-View Surface Reconstruction of Large-Scale Scenes

“…Yang and Jiang [42] combine deep learning algorithms with traditional methods to extract and match feature points from light pattern augmented images to improve a practical 3D reconstruction method for weakly textured scenes. Stathopoulou et al [43] tackle the textureless problem by leveraging semantic priors into a PatchMatch-based MVS in order to increase confidence and better support depth and normal map estimation on weakly textured areas. However, even with these combination of traditional and learning algorithms, visual reconstruction of large textureless areas commonly present in urban scenarios of building facades or indoor scenes still remains to be a challenge.…”