An Image-Based Deep Learning Workflow for 3d Heritage Point Cloud Semantic Segmentation

Abstract

Abstract. The interest in high-resolution semantic 3D models of historical buildings continuously increased during the last decade, thanks to their utility in protection, conservation and restoration of cultural heritage sites. The current generation of surveying tools allows the quick collection of large and detailed amount of data: such data ensure accurate spatial representations of the buildings, but their employment in the creation of informative semantic 3D models is still a challenging task, and it currently still requires manual time-consuming intervention by expert operators. Hence, increasing the level of automation, for instance developing an automatic semantic segmentation procedure enabling machine scene understanding and comprehension, can represent a dramatic improvement in the overall processing procedure. In accordance with this observation, this paper aims at presenting a new workflow for the automatic semantic segmentation of 3D point clouds based on a multi-view approach. Two steps compose this workflow: first, neural network-based semantic segmentation is performed on building images. Then, image labelling is back-projected, through the use of masked images, on the 3D space by exploiting photogrammetry and dense image matching principles. The obtained results are quite promising, with a good performance in the image segmentation, and a remarkable potential in the 3D reconstruction procedure.

Instant Segmentation and Fitting of Excavations in Subsurface Utility Engineering

“…Segmentation in 2D Some segmentation algorithms [34,37] use deep learning to create segmentation masks, which are then backprojected [22] to the 3D point cloud. Such virtual views have been used by Kundus et al [27] to segment 3D meshes.…”

3d Documentation Towards Heritage Conservation: A Rapid Mapping Approach Applied to Bahrain Forts

“…Usually, the masonry texture is one the fundamental information that can be derived and reconstructed in the 2D and 3D representation. Nowadays semi-automatic process of vectorization and classification based on Machine Learning (ML) algorithms are available (Wang et al 2021;Pellis et al 2022) and they can be further tested based on these datasets (see Conclusion). To apply these semi-automatic approaches, the geometric characteristics of the surface and the radiometric contrast must be optimal for applying pixel-based or point-based strategies or integrating them.…”

Point-Cloud Segmentation for 3D Edge Detection and Vectorization

“…Obviously, the proposed approach requires the generation of a dense point cloud, which is a time-consuming process, twice to produce the true colors and the semantically enriched point clouds. Pellis et al [2] proposed a new workflow for 3D point-cloud semantic segmentation for cultural-heritage buildings, using photogrammetric and dense image-matching principles. The proposed workflow contains two major steps: firstly, the images are semantically segmented using DeepLabv3+ and secondly, the semantic information is projected into 3D spaces exploiting image masks.…”

“…A plethora of methods developed so far dealing with 3D point-cloud semantic segmentation exploit various transformations of the cloud such as graphs and voxels or are applied directly on it. Furthermore, deep-learning image semantic-segmentation algorithms are combined with the photogrammetric pipeline to produce semantically rich 3D point clouds [1][2][3]. Additionally, 2D-3D semantic-segmentation techniques are exploited to improve other methods such as 3D reconstruction [4,5].…”