2d to 3d Label Propagation for the Semantic Segmentation of Heritage Building Point Clouds

Abstract

Abstract. During the last decade, the use of semantic models of 3D buildings and structures kept growing, fostered in particular by the spread of Building Information Models (BIMs), becoming quite popular in several civil engineering and geomatics applications. Nevertheless, semantic model production usually requires quite a lot of human interaction, which may result in quite long and annoying procedures for human operators. The production of 3D semantic models of buildings often takes advantage of already available 3D reconstructions of the considered objects. Given the ever increasing resolution of 3D reconstructions, obtained thanks to the recently developed laser scanners and photogrammetric software, the availability of tools for supporting the automatic or semi-automatic generation of semantic models represents a key step for easing and speeding up the process of semantic model production. In particular, the correct semantic interpretation of the different parts of a 3D point cloud, can be seen as the basic step for the production of a BIM model. The most frequently used methods for point cloud semantic segmentation can be separated in two categories: those directly segmenting the point clouds and those based on the ancillary semantic segmentation of images representing the object of interest, then transferring back the segmentation results to the point cloud. This work focuses on the latter method, considering more specifically the application of heritage building semantic segmentation. To be more specific, this paper investigates the semantic segmentation performance on a set of four heritage buildings, obtained first applying deep-learning based image semantic segmentation and then propagating back the semantic information to the point cloud by means of a voting strategy. The obtained results are quite encouraging, motivating future investigations on improvements of this strategy, in particular when including more buildings in the considered dataset.

A Performance Comparison Between Segnet and Deeplabv3+ on the Semantic Segmentation of Heritage Buildings

“…Secondly, the results on point cloud segmentation are illustrated. They are the outcomes of the labelling projection procedure introduced in (Pellis et al, 2022b). For each test the GA, the mIoU and the confusion matrices and shown, together with some predicted segmentation maps.…”

Machine and Deep Learning Implementations for Heritage Building Information Modelling: A Critical Review of Theoretical and Applied Research

“…Even more impressive results can be found in a study by [80], where a four-layer CNN process applied directly to 3D point cloud input data achieved 92.4% accuracy. In the context of 2.5D CNN methods applied to Cultural Heritage datasets, [81] proposed a pioneering method that achieves depth-based image classification from the ArCH benchmark dataset and applies a feature fusion process to classify the complementary point clouds. The results of this method achieved a global accuracy of 87%-90% and are presented in Figure 12.…”

Material augmented semantic segmentation of point clouds for building elements

“…These images can be subjected to semantic segmentation, and predicted labels in images are then mapped to corresponding 3D points. The resulting mapped labels are further refined to generate segmentation results, which are used for the volumetric measurement of stockpiles (Kamari & Ham, 2021), segmentation of heritage buildings (Pellis et al., 2022), and extraction of steel structures (Smith & Sarlo, 2022). Although these methods succeed in regularizing point cloud data for use in CNN models, the conversion of 3D point clouds and remapping of 2D labels result in additional loss of fine details in the original point cloud during the process.…”