Generalized Sparse Convolutional Neural Networks for Semantic Segmentation of Point Clouds Derived from Tri-Stereo Satellite Imagery

Abstract

We studied the applicability of point clouds derived from tri-stereo satellite imagery for semantic segmentation for generalized sparse convolutional neural networks by the example of an Austrian study area. We examined, in particular, if the distorted geometric information, in addition to color, influences the performance of segmenting clutter, roads, buildings, trees, and vehicles. In this regard, we trained a fully convolutional neural network that uses generalized sparse convolution one time solely on 3D geometric information (i.e., 3D point cloud derived by dense image matching), and twice on 3D geometric as well as color information. In the first experiment, we did not use class weights, whereas in the second we did. We compared the results with a fully convolutional neural network that was trained on a 2D orthophoto, and a decision tree that was once trained on hand-crafted 3D geometric features, and once trained on hand-crafted 3D geometric as well as color features. The decision tree using hand-crafted features has been successfully applied to aerial laser scanning data in the literature. Hence, we compared our main interest of study, a representation learning technique, with another representation learning technique, and a non-representation learning technique. Our study area is located in Waldviertel, a region in Lower Austria. The territory is a hilly region covered mainly by forests, agriculture, and grasslands. Our classes of interest are heavily unbalanced. However, we did not use any data augmentation techniques to counter overfitting. For our study area, we reported that geometric and color information only improves the performance of the Generalized Sparse Convolutional Neural Network (GSCNN) on the dominant class, which leads to a higher overall performance in our case. We also found that training the network with median class weighting partially reverts the effects of adding color. The network also started to learn the classes with lower occurrences. The fully convolutional neural network that was trained on the 2D orthophoto generally outperforms the other two with a kappa score of over 90% and an average per class accuracy of 61%. However, the decision tree trained on colors and hand-crafted geometric features has a 2% higher accuracy for roads.

Methods in the spatial deep learning: current status and future direction

“…Segmentation operations with point clouds were performed directly on 3D points rather than on projected surfaces or voxels. Bachhofner et al (2020) used U-Net architecture-based generalized sparse convolutional neural network (GSCNN) built with sparse convolution blocks to segment 3D points generated from tri-stereo satellite imagery [85]. The segmentation of archeological and cultural heritage sites was performed with a dynamic graph convolution neural network (DGCNN) constructed with several blocks of edge convolutional layers [74].…”

Comparison of Graph Fitting and Sparse Deep Learning Model for Robot Pose Estimation

“…Sparse signals are a natural representation of images obtained from depth sensors. Using sparse layers and inputs in CNNs is not as popular as dense ones, but it has also been considered (e.g., [22,23]). Some kinds of sparsity are available in commonly used deep learning libraries.…”

Supervised Classification and Its Repeatability for Point Clouds From Dense VHR Tri-Stereo Satellite Image Matching Using Machine Learning

“…In addition to the hand-crafted 3D geometric features, the RGB colour information in each point was used for training the classifier. For the model learning process we used a total of 17 features: Red, Green, Blue, EchoNumber, Number of echos, Amplitude, Normal X, Normal Y, Normal Z, Normal sigma0, linearity, planarity, omnivariance, EchoRatio, NormalizedZ, dZRange, and dZRank (Bachhofner et al, 2020;Waldhauser et al, 2014). Like all supervised classification methods, the adopted decision tree requires training data, which are used by the machine learning algorithm to build the classification model.…”

“…The classification tree seeks to partition the entire feature space of a data set, one variable at a time, by selecting a variable and an appropriate splitting value (Waldhauser et al, 2014). The decision tree is trained with the following hyperparameters: 0.00001 complexity factor, minimum 20 observations existent in a node (for splitting), at least 7 observations for each leaf node, a maximum depth of 30 for the tree, 5 competitor splits, and 5 surrogate splits (Bachhofner et al, 2020). Finally, to estimate how accurately our predictive model is, we test it on the validation dataset, containing labelled points.…”