Unsupervised Domain Adaptation with Transformer-Based GAN for Semantic Segmentation of High-Resolution Remote Sensing Images

Abstract

<p>Cross-domain semantic segmentation of remote sensing (RS) imagery based on unsupervised domain adaptation (UDA) has become a research hotspot in geoscience research. Recently, transformer, with its ingenious and versatile architecture, has been successfully applied in a wide range of RS tasks. Despite some attempts to integrate transformers with convolutional neural networks (CNNs) in UDA, existing works appear ineffective at leveraging transformer structures, which is evidenced by the large performance gap between these existing UDA methods and supervised learning-based methods. In this work, we adopt a radical and subversive stance by completely overhauling the UDA framework using transformer structures to bridge the gap across different domains based on generative adversarial networks (GAN). To this end, we first present a transformer-based generator (TransG) consisting of two components, namely a transformer-based encoder (TransEn) and a transformer-based decoder (TransDe) equipped with global-local transformer blocks (GLTB). The TransEn can extract contextual details from high-resolution RS images more efficiently as compared to CNN-based encoders, whereas the TransDe restores accurate spatial and local details simultaneously during the segmentation process. Furthermore, a novel transformer-based discriminator (TransDx) is proposed to encourage TransG to learn better domain-invariant representations. The TransG and TransDx together constitute the proposed transformer-based GAN, named DA-TransGAN, which provides an easily comprehensible UDA framework that can be extended to general UDA tasks in a straightforward manner. Extensive experiments on two large-scale fine-resolution benchmark datasets, namely ISPRS Potsdam and Vaihingen, highlight the effectiveness and superiority of the proposed DA-TransGAN as compared to the state-of-the-art UDA methods.</p>

Enhancing precipitation prediction accuracy with Transformer-GAN hybrid models

“…Generative Adversarial Networks (GANs) have been increasingly applied in the field of meteorology to address a range of challenges. One of the most notable applications is in data augmentation, where GANs are used to generate synthetic weather patterns to supplement existing datasets, thereby improving model training and validation [1] . Another significant application is in downscaling coarse-grained weather forecasts to produce high-resolution, localized predictions.…”

“…Adversarial Loss (Ladv): Used to train the GAN module, this loss ensures that the Discriminator is unable to distinguish between real and generated precipitation maps. The total loss is then given by: L=αLcontent+βLadv+γLfeature (1) where α, β, andγ are hyperparameters that control the contribution of each component to the total loss.…”