Semantic Enrichment of Point Cloud by Automatic Extraction and Enhancement of 360° Panoramas

Abstract

Abstract. The raw nature of point clouds is an important challenge for their direct exploitation in architecture, engineering and construction applications. Particularly, their lack of semantics hinders their utility for automatic workflows (Poux, 2019). In addition, the volume and the irregularity of the structure of point clouds makes it difficult to directly and automatically classify datasets efficiently, especially when compared to the state-of-the art 2D raster classification. Recently, with the advances in deep learning models such as convolutional neural networks (CNNs) , the performance of image-based classification of remote sensing scenes has improved considerably (Chen et al., 2018; Cheng et al., 2017). In this research, we examine a simple and innovative approach that represent large 3D point clouds through multiple 2D projections to leverage learning approaches based on 2D images. In other words, the approach in this study proposes an automatic process for extracting 360° panoramas, enhancing these to be able to leverage raster data to obtain domain-base semantic enrichment possibilities. Indeed, it is very important to obtain a rigorous characterization for use in the classification of a point cloud. Especially because there is a very large variety of 3D point cloud domain applications. In order to test the adequacy of the method and its potential for generalization, several tests were performed on different datasets. The developed semantic augmentation algorithm uses only the attributes X, Y, Z and camera positions as inputs.

3D Point Cloud Semantic Augmentation: Instance Segmentation of 360° Panoramas by Deep Learning Techniques

“…Thus, we preserve the richness of the initial point cloud in its entirety. In the second row, we notice that the upper and lower facet were deformed due to the lack of data at floor and ceiling level in the panoramic image generated by Tabkha (2019) [20]. On the other hand, the cubemap generated by our approach gives a geometrically correct image.…”

“…We organise the related works into the two following sections: Section 2.1 includes the definition of panoramic images and geometric projections that encompass a big part of the current research; Section 2.2 describes the different deep learning semantic extraction techniques in computer vision. In this same section, we give a brief cover of Tabkha's thesis [20], which preceded this research.…”

“…To give a chronological order, this research follows Tabkha's thesis [20], which was elaborated in 2019 under the title "Semantic Increase of a Point Cloud through the Automatic Extraction of a 360 • Panorama and Its Integration into a Platform of Machine Learning ".…”

Semantic Segmentation of Terrestrial Lidar Data Using Co-Registered RGB Data

“…These methods are typically a lot faster than purely 3D approaches as projection and 2D convolutions are much faster than 3D neighbourhood searches required by geometric-based approaches. Similar to our work, Tabkha et al (2019) perform semantic segmentation using a Convolutional Neural Network (CNN) on RGB images derived by projecting coloured 3D point clouds. However, our work differs from these approaches as we do not use an unordered point cloud as the representation for the LiDAR data.…”

Initial User-Centered Design of a Virtual Reality Heritage System: Applications for Digital Tourism

“…Finally, a representation is proposed to communicate through a medium such as a web environment, a 3D simulation software, or, more basically, by producing 2D documents such as maps and plans. Some in-depth information can be found in [48,49]. A dichotomy still exists in the documentation that the virtual environment can use to reproduce the cultural heritage sites [50].…”