Multi-View Stereo With Semantic Priors

Abstract

<p><strong>Abstract.</strong> Patch-based stereo is nowadays a commonly used image-based technique for dense 3D reconstruction in large scale multi-view applications. The typical steps of such a pipeline can be summarized in stereo pair selection, depth map computation, depth map refinement and, finally, fusion in order to generate a complete and accurate representation of the scene in 3D. In this study, we aim to support the standard dense 3D reconstruction of scenes as implemented in the open source library OpenMVS by using semantic priors. To this end, during the depth map fusion step, along with the depth consistency check between depth maps of neighbouring views referring to the same part of the 3D scene, we impose extra semantic constraints in order to remove possible errors and selectively obtain segmented point clouds per label, boosting automation towards this direction. In order to reassure semantic coherence between neighbouring views, additional semantic criterions can be considered, aiming to eliminate mismatches of pixels belonging in different classes.</p>

Surface Reconstruction Assessment in Photogrammetric Applications

“…Based on PatchMatch stereo [71,72], Shen [24] introduced a patch-based stereo approach where the depth of each pixel is calculated using random assignment and spatial propagation. OpenMVS is an open-source library that closely follows this idea while applying some optimization steps for more efficiency and is thus broadly used in 3D reconstruction research [42,73,74]. First, the best neighboring views are selected based on viewing direction criteria, and potential stereo pairs are formed.…”

3D Reconstruction of a Complex Grid Structure Combining UAS Images and Deep Learning

“…CNN methods [53][54][55][56][57][58][59] have also demonstrated their potential for detecting a numerous number of elements in the images and then boost the processing pipeline in terms of constrained tie point extraction or semantic multi-view stereo [60][61][62]. The advantages of image masking for dense point cloud generation are well known in the literature [62][63][64].…”

“…CNN methods [53][54][55][56][57][58][59] have also demonstrated their potential for detecting a numerous number of elements in the images and then boost the processing pipeline in terms of constrained tie point extraction or semantic multi-view stereo [60][61][62]. The advantages of image masking for dense point cloud generation are well known in the literature [62][63][64]. While there are multiple readily available segmenation models for oblique aerial photos [63] or buildings [64,65], the generation of pixel-level semantic segmentation for sparse wire objects is challenging.…”

“…The advantages of image masking for dense point cloud generation are well known in the literature [62][63][64]. While there are multiple readily available segmenation models for oblique aerial photos [63] or buildings [64,65], the generation of pixel-level semantic segmentation for sparse wire objects is challenging. The analysis of repetitive patterns [66,67] allows to partly solve this problem for opaque objects (e.g., skyscrapers).…”

“…Therefore, manual labelling of all acquired data was impossible. The presented approach was inspired by recent research [62] which used semantic segmentation in images for improving accuracy of a multi-view stereo processing. A deep learning based technique is proposed to automatically generate image masks in case of complex wired structures (like the Shukhov tower).…”