Towards Semantic Photogrammetry: Generating Semantically Rich Point Clouds from Architectural Close-Range Photogrammetry

Abstract

Developments in the field of artificial intelligence have made great strides in the field of automatic semantic segmentation, both in the 2D (image) and 3D spaces. Within the context of 3D recording technology it has also seen application in several areas, most notably in creating semantically rich point clouds which is usually performed manually. In this paper, we propose the introduction of deep learning-based semantic image segmentation into the photogrammetric 3D reconstruction and classification workflow. The main objective is to be able to introduce semantic classification at the beginning of the classical photogrammetric workflow in order to automatically create classified dense point clouds by the end of the said workflow. In this regard, automatic image masking depending on pre-determined classes were performed using a previously trained neural network. The image masks were then employed during dense image matching in order to constraint the process into the respective classes, thus automatically creating semantically classified point clouds as the final output. Results show that the developed method is promising, with automation of the whole process feasible from input (images) to output (labelled point clouds). Quantitative assessment gave good results for specific classes e.g., building facades and windows, with IoU scores of 0.79 and 0.77 respectively.

2d to 3d Label Propagation for the Semantic Segmentation of Heritage Building Point Clouds

“…The introduction of an additional step in the label propagation procedure, in order to ensure the classification of most the currently unlabelled points, will be considered in our future works, as previously mentioned. Finally, for comparison with the results presented here, Table 6 shows those obtained with the masking-based methodology described in (Murtiyoso et al, 2022) and carried out in . Despite the masking method works well for background removal and for building façade classification, the label propagation method considered here outperforms the masking-based one on our dataset, highlighting the still challenging use of semantically enriched reconstruction methods in complex scenarios.…”

“…The XY coordinates of each pixel in the orthophoto was used to determine the corresponding planimetric coordinates of the point in the point cloud and finally a winner-takes-all approach was applied to annotate the 3D points with the respective 2D pixel class. In a more recent work (Murtiyoso et al, 2022) introduced semantic classification at the beginning of the classical photogrammetric workflow in order to automatically create a classified dense point cloud. In this regard, several image masks obtained by a trained neural network are employed during dense image matching in order to constraint the process into the respective classes.…”

A Perspective on AI-Based Image Analysis and Utilization Technologies in Building Engineering: Recent Developments and New Directions

“…A noteworthy project is that of semantic segmentation which aims to enhance the photogrammetric pipeline by integrating semantic information within the processing phases. In [12], an approach to introduce AI-based semantic segmentation in the photogrammetric workflow was presented. The proposed workflow uses 2D image label data and robust AI-based methods to create separate point clouds for each class, demonstrating that the assumption of using the far more available labeled 2D training data is beneficial.…”

Comparison of Deep Learning Architectures for the Semantic Segmentation of Slum Areas From Satellite Images

“…The use of AI (artificial intelligence) in classifying satellite images has seen a recent surge in interest. In AI parlance, the act of classifying pixels is analogous to semantic segmentation (Murtiyoso et al, 2022), although it may also involve instance segmentation and eventually panoptic segmentation (Kirillov et al, 2019). Slum area classification using deep learning approach had been conducted by Mboga et al (2017); Persello and Stein (2017); Gram-Hansen et al (2019); Liu et al (2019); Fisher et al (2022).…”