Joint alignment of the distribution in input and feature space for cross-domain aerial image semantic segmentation

Chen, Zhe; Wang, Jinhu; Ma, Ailong; Peng, Mingjun; Li, Haiting; Chen, Tao; Chen, Chi; Dong, Zhen

doi:10.1016/j.jag.2022.103107

“…In remote sensing, ToMF-B [63] decomposed the crossdomain features into task-related features and task-irrelevant features by analyzing the gradients of the predicted score corresponding to the labels. ST-DASegNet [64] introduced domain disentangled module to extract cross-domain universal features and purify single-domain distinct features in a selftraining guided framework. Considering the current research status in remote sensing, the decomposition-based methods in the feature extraction stage to promote cross-domain feature alignment still needs to be explored.…”

Section: B Decomposition-based Unsupervised Domain Adaptationmentioning

confidence: 99%

Unsupervised Domain Adaptation with Transformer-Based GAN for Semantic Segmentation of High-Resolution Remote Sensing Images

Ma¹,

Ding²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Cross-domain semantic segmentation of remote sensing (RS) imagery based on unsupervised domain adaptation (UDA) has become a research hotspot in geoscience research. Recently, transformer, with its ingenious and versatile architecture, has been successfully applied in a wide range of RS tasks. Despite some attempts to integrate transformers with convolutional neural networks (CNNs) in UDA, existing works appear ineffective at leveraging transformer structures, which is evidenced by the large performance gap between these existing UDA methods and supervised learning-based methods. In this work, we adopt a radical and subversive stance by completely overhauling the UDA framework using transformer structures to bridge the gap across different domains based on generative adversarial networks (GAN). To this end, we first present a transformer-based generator (TransG) consisting of two components, namely a transformer-based encoder (TransEn) and a transformer-based decoder (TransDe) equipped with global-local transformer blocks (GLTB). The TransEn can extract contextual details from high-resolution RS images more efficiently as compared to CNN-based encoders, whereas the TransDe restores accurate spatial and local details simultaneously during the segmentation process. Furthermore, a novel transformer-based discriminator (TransDx) is proposed to encourage TransG to learn better domain-invariant representations. The TransG and TransDx together constitute the proposed transformer-based GAN, named DA-TransGAN, which provides an easily comprehensible UDA framework that can be extended to general UDA tasks in a straightforward manner. Extensive experiments on two large-scale fine-resolution benchmark datasets, namely ISPRS Potsdam and Vaihingen, highlight the effectiveness and superiority of the proposed DA-TransGAN as compared to the state-of-the-art UDA methods.</p>

show abstract

“…In remote sensing, ToMF-B [63] decomposed the crossdomain features into task-related features and task-irrelevant features by analyzing the gradients of the predicted score corresponding to the labels. ST-DASegNet [64] introduced domain disentangled module to extract cross-domain universal features and purify single-domain distinct features in a selftraining guided framework. Considering the current research status in remote sensing, the decomposition-based methods in the feature extraction stage to promote cross-domain feature alignment still needs to be explored.…”

Section: B Decomposition-based Unsupervised Domain Adaptationmentioning

confidence: 99%

Decomposition-based Unsupervised Domain Adaptation for Remote Sensing Images Semantic Segmentation

Ma¹,

Ding²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Cross-domain semantic segmentation of remote sensing (RS) imagery based on unsupervised domain adaptation (UDA) has become a research hotspot in geoscience research. Recently, transformer, with its ingenious and versatile architecture, has been successfully applied in a wide range of RS tasks. Despite some attempts to integrate transformers with convolutional neural networks (CNNs) in UDA, existing works appear ineffective at leveraging transformer structures, which is evidenced by the large performance gap between these existing UDA methods and supervised learning-based methods. In this work, we adopt a radical and subversive stance by completely overhauling the UDA framework using transformer structures to bridge the gap across different domains based on generative adversarial networks (GAN). To this end, we first present a transformer-based generator (TransG) consisting of two components, namely a transformer-based encoder (TransEn) and a transformer-based decoder (TransDe) equipped with global-local transformer blocks (GLTB). The TransEn can extract contextual details from high-resolution RS images more efficiently as compared to CNN-based encoders, whereas the TransDe restores accurate spatial and local details simultaneously during the segmentation process. Furthermore, a novel transformer-based discriminator (TransDx) is proposed to encourage TransG to learn better domain-invariant representations. The TransG and TransDx together constitute the proposed transformer-based GAN, named DA-TransGAN, which provides an easily comprehensible UDA framework that can be extended to general UDA tasks in a straightforward manner. Extensive experiments on two large-scale fine-resolution benchmark datasets, namely ISPRS Potsdam and Vaihingen, highlight the effectiveness and superiority of the proposed DA-TransGAN as compared to the state-of-the-art UDA methods.</p>

show abstract

“…Directly acquired point cloud data lack auxiliary information, so categorizing each point and providing semantic information is necessary for the effective performance of the subsequent relative tasks. Currently, image semantic segmentation [15], change detection [16][17][18], and classification [19] have achieved significant success due to the improvement of deep learning. These achievements have spurred research towards the effective application of deep learning to point cloud tasks [20], which has become an important research direction.…”

Section: Introductionmentioning

confidence: 99%

BEMF-Net: Semantic Segmentation of Large-Scale Point Clouds via Bilateral Neighbor Enhancement and Multi-Scale Fusion

Ji,

Yang,

Jiang

et al. 2023

Remote Sensing

0

View full text Add to dashboard Cite

The semantic segmentation of point clouds is a crucial undertaking in 3D reconstruction and holds great importance. However, achieving precise semantic segmentation represents a significant hurdle. In this paper, we present BEMF-Net, an efficient method for large-scale environments. It starts with an effective feature extraction method. Unlike images, 3D data comprise not only geometric relations but also texture information. To accurately depict the scene, it is crucial to take into account the impacts of texture and geometry on the task, and incorporate modifications to improve feature description. Additionally, we present a multi-scale feature fusion technique that effectively promotes the interaction between features at different resolutions. The approach mitigates the problem of the smoothing of detailed information caused by downsampling mechanisms, while ensuring the integrity of features across different layers, allowing a more comprehensive representation of the point cloud. We confirmed the effectiveness of this method by testing it on benchmark datasets such as S3DIS, SensatUrban, and Toronto3D.

show abstract

Joint alignment of the distribution in input and feature space for cross-domain aerial image semantic segmentation

Cited by 5 publications

References 33 publications

Unsupervised Domain Adaptation with Transformer-Based GAN for Semantic Segmentation of High-Resolution Remote Sensing Images

Unsupervised Domain Adaptation with Transformer-Based GAN for Semantic Segmentation of High-Resolution Remote Sensing Images

Decomposition-based Unsupervised Domain Adaptation for Remote Sensing Images Semantic Segmentation

BEMF-Net: Semantic Segmentation of Large-Scale Point Clouds via Bilateral Neighbor Enhancement and Multi-Scale Fusion

Contact Info

Product

Resources

About