Comparison of 2D and 3D semantic segmentation in urban areas using fused hyperspectral and lidar data

Abstract

Multisensor data fusion has become a hot topic in the remote sensing research community. This is thanks to significant technological advances and the ability to extract information that would have been challenging with a single sensor. However, sensory enhancement requires advanced analysis that enables deep learning. A framework is designed to effectively fuse hyperspectral and lidar data for semantic segmentation in the urban environment. Our work proposes a method of reducing dimensions by exploring the most representative features from hyperspectral and lidar data and using them for supervised semantic segmentation. In addition, we chose to compare segmentation models based on 2D and 3D convolutional operations with two different model architectures, such as U-Net and ResU-Net. All algorithms have been tested with three loss functions: standard Categorical Cross-Entropy, Focal Loss and a combination of Focal Loss and Jaccard Distance—Focal–Jaccard Loss. Experimental results demonstrated that the 3D segmentation of U-Net and ResU-Net with Focal and Focal–Jaccard Loss functions had significantly improved performance compared to the standard Categorical Cross-Entropy models. The results show a high accuracy score and reflect reality by preserving the complex geometry of the objects.

Multitemporal Feature-Level Fusion on Hyperspectral and LiDAR Data in the Urban Environment

“…Ground truth data were unavailable for low and high vegetation due to high dynamic and seasonal differences. Therefore, these classes were extracted semi-automatically, calculating the Normalised Difference Vegetation Index (NDVI) for the HS scene [48] and distinguished high and low vegetation based on raster-based LiDAR features, relying on the method from Kuras et al [35] (Figure 1d). The main features for differentiating high and low vegetation were selected prior to analysis based on knowledge and experience.…”

“…The final input to semantic segmentation algorithms consists of abundance maps from HS and LiDAR data. We considered the 2D ResU-Net model architectures in this study [35,53,54], comparing the segmentation process with and without training data augmentation for 2019 (Figure 2, box 3) and 2021 (Figure 2, box 6) datasets without model parameter regularizations. The original U-Net consists of an encoder part with multiple blocks of convolutions and max pools for feature extraction and a corresponding decoder with transposed convolutions for upscaling after each convolution block [59].…”

“…The Residual U-Net extends the original U-Net with local skip connections in the convolution blocks, further enhancing signal pathways and granularity of predictions. In the ResU-Net model, we implemented 2D convolutional operations, which are sufficient and not time-consuming in this type of land cover analysis [35].…”

“…Airborne campaigns are being launched increasingly in which data from different sensors are acquired simultaneously, such as RGB cameras, LiDAR scanners, and multispectral and HS sensors [35][36][37][38][39][40][41]. This opens up the possibility to fuse data from different sensors at different levels ranging from raw data fusion, feature-level fusion, or application-level fusion [42].…”

“…Khodadadzadeh et al [50] proposed a feature-level fusion method integrating multiple types of HS and LiDAR-based features without model parameter regularization. Kuras et al [35] extracted endmembers from an HS dataset in an unsupervised way by applying N-FINDR [51] and retrieving raster-based LiDAR features for segmentation purposes.…”