Towards Mesh-Based Deep Learning for Semantic Segmentation in Photogrammetry

Abstract

Abstract. This research is the first to apply MeshCNN – a deep learning model that is specifically designed for 3D triangular meshes – in the photogrammetry domain. We highlight the challenges that arise when applying a mesh-based deep learning model to a photogrammetric mesh, especially w.r.t. data set properties. We provide solutions on how to prepare a remotely sensed mesh for a machine learning task. The most notable pre-processing step proposed is a novel application of the Breadth-First Search algorithm for chunking a large mesh into computable pieces. Furthermore, this work extends MeshCNN such that photometric features based on the mesh texture are considered in addition to the geometric information. Experiments show that including color information improves the predictive performance of the model by a large margin. Besides, experimental results indicate that segmentation performance could be advanced substantially with the introduction of a high-quality benchmark for semantic segmentation on meshes.

Semantic Segmentation of Urban Textured Meshes Through Point Sampling

“…MeshCNN (Hanocka et al, 2019) introduce a mesh-specific convolution and pooling layers that are applied over the edges of a mesh. Knott and Groenendijk (2021) add radiometric features to face inspired by Rouhani et al (2017).…”

MeshNet-SP: A Semantic Urban 3D Mesh Segmentation Network with Sparse Prior

“…Voxel-based methods [22][23][24][25][26] convert the irregular 3D meshes into regular 3D grids, i.e., 3D voxels, and then 3D CNNs are applied to these 3D voxels. Different from CoGP-based and voxel-based methods, meshbased methods [27][28][29][30][31][32][33][34][35] directly perform convolution on 3D meshes, using the topological information of vertices/edges/facets within 3D meshes. In view-based methods [36][37][38], different virtual views of the 3D mesh were used to render multiple 2D channels for training an effective 2D semantic segmentation model.…”