Assembling an Image and Point Cloud Dataset for Heritage Building Semantic Segmentation

Abstract

Abstract. Creating three-dimensional as-built models from point clouds is still a challenging task in the Cultural Heritage environment. Nowadays, performing such task typically requires the quite time-consuming manual intervention of an expert operator, in particular to deal with the complexities and peculiarities of heritage buildings. Motivated by these considerations, the development of automatic or semi-automatic tools to ease the completion of such task has recently became a very hot topic in the research community. Among the tools that can be considered to such aim, the use of deep learning methods for the semantic segmentation and classification of 2D and 3D data seems to be one of the most promising approaches. Indeed, these kinds of methods have already been successfully applied in several applications enabling scene understanding and comprehension, and, in particular, to ease the process of geometrical and informative model creation. Nevertheless, their use in the specific case of heritage buildings is still quite limited, and the already published results not completely satisfactory. The quite limited availability of dedicated benchmarks for the considered task in the heritage context can also be one of the factors for the not so satisfying results in the literature.Hence, this paper aims at partially reducing the issues related to the limited availability of benchmarks in the heritage context by presenting a new dataset for semantic segmentation of heritage buildings. The dataset is composed by both images and point clouds of the considered buildings, in order to enable the implementation, validation and comparison of both point-based and multiview-based semantic segmentation approaches. Ground truth segmentation is provided, for both the images and point clouds related to each building, according to the class definition used in the ARCHdataset, hence potentially enabling also the integration and comparison of the results obtained on such dataset.

A Performance Comparison Between Segnet and Deeplabv3+ on the Semantic Segmentation of Heritage Buildings

“…The dataset developed in (Pellis et al, 2021) has been used to compare the two neural network models. The dataset is composed by five heritage buildings, and for each building three types of data are available: (i) the laser point cloud, (ii) the photogrammetric images and (iii) the related point clouds.…”

2d to 3d Label Propagation for the Semantic Segmentation of Heritage Building Point Clouds

“…To test the proposed procedure, the dataset presented in (Pellis et al, 2021), currently composed by three heritage buildings, has been used. For each building several data types are available: (i) the Terrestrial Laser Scanner (TLS) cloud with the corresponding ground-truth segmentation, (ii) the photogrammetric cloud with the corresponding ground-truth segmentation, and (iii) the RGB images of the photogrammetric survey with the corresponding pixel-wise ground-truth segmentation.…”

“…We used a pretrained version of the network on the ImageNet database (Deng et al, 2009) with ResNet-18 (He et al, 2015) as base classification architecture. The testing dataset (Pellis et al, 2021) is still in progress, and it still lacks of a sufficient variability in the images and building typologies to well-generalize a complete unseen scenario. Nevertheless, some tests, varying the complexity of the goal, have been performed to check the label prediction ability of the network, as shown below.…”

“…This paper takes advantage of a previously developed image-semantic segmentation network (Pellis et al, 2022), and it focuses on the development of a reliable 2D to 3D label transfer procedure with the main aim of decreasing the geometric and spatial information loss and improving the overall accuracy of the image-based semantic segmentation workflow. In this paper the procedure will be tested on the case of heritage buildings scenarios, using an ongoing dataset (Pellis et al, 2021) specifically suited for heritage building semantic segmentation.…”