Homo–Heterogenous Transformer Learning Framework for RS Scene Classification

Abstract

Remote sensing (RS) scene classification plays an essential role in the RS community and has attracted increasing attention due to its wide applications. Recently, benefiting from the powerful feature learning capabilities of convolutional neural networks (CNNs), the accuracy of RS scene classification has significantly been improved. Although the existing CNNbased methods achieve excellent results, there is still room for improvement. First, the CNN-based methods are adept at capturing the global information from RS scenes. Still, the context relationships hidden in RS scenes cannot be thoroughly mined. Second, due to the specific structure, it is easy for normal CNNs to exploit the heterogenous information from RS scenes. Nevertheless, the homogenous information, which is also crucial to comprehensively understand complex contents within RS scenes, does not get the attention it deserves. Third, most CNNs focus on establishing the relationships between RS scenes and semantic labels. However, the similarities between them are not considered deeply, which are helpful to distinguish the intra-/inter-class samples. To overcome the limitations mentioned above, we propose a homo-heterogenous transformer learning (HHTL) framework for RS scene classification in this paper. First, a patch generation module (PGM) is designed to generate homogenous and heterogenous patches. Then, a dual-branch feature learning module (FLM) is proposed to mine homogenous and heterogenous information within RS scenes simultaneously. In FLM, based on vision transformer, not only the global information but also the local areas and their context information can be captured. Finally, we design a classification module, which consists of a fusion sub-module and a metric-learning module. It can integrate homo-heterogenous information and compact/separate samples from the same/different RS scene categories. Extensive experiments are conducted on four public RS scene data sets. The encouraging results demonstrate that our HHTL framework can outperform many state-of-the-art methods. Our source codes are available at https://github.com/TangXu-Group/Remote-Sensing-Images-Classification/tree/main/HHTL.

Deep learning techniques for remote sensing image scene classification: A comprehensive review, current challenges, and future directions

“…Lu et al 62 Explores the content of semantic label Li et al 111 Captures the key region Bi et al 110 Direct control over semantic labels at the bag level Li et al 111 Reduces the dimensionality of bilinear pooled features Wang et al 112 Addresses the problem of large scale variance Xu et al 76 Extracts the context information Xu et al 114 Improves discriminative ability among ambiguous classes Deng et al 63 Extracts semantic features and local structural features Ma et al 75 Captures contextual relationship in complex RSI AB Wang et al 115 Reduces the number of training parameters Li et al 116 Fuses global and local features.…”

“…Zhao et al 118 Extracts stronger discriminative features Li et al 119 Eliminates irrelevant and redundant information Tang et al 120 Minimize intra class distance and maximize inter class distance Bi et al 121 Local semantic representation capability is enhanced Chen et al 122 Improves feature extraction ability GAN Lin et al 98 Captures powerful discriminative features Yu et al 106 Collects contextual information Wei et al 105 Multi feature fusion layer is incorporated Ma et al 75 Generates labeled samples Miao et al 107 Deals with overfitting problem of deep models SOSFB He et al 61 Preserves second order information.…”

Transformer based on channel-spatial attention for accurate classification of scenes in remote sensing image

“…Based on the basic CNN, CTNet 30 delicately develops an enhanced version of the CNNbased network. HHTL 31 carefully designed the patch before it was input to the transformers and subtly fused them after feature extraction. Some methods improved the classification performance by adding operations, e.g., multi-scale, spatial attention, and feature aggregation.…”

“…Therefore, Li et al 30 proposed the remote sensing transformer (TRS), which uses self-attention integrated into a residual neural network (ResNet), Multi-Head Self-Attention (MHSA) layers instead of spatial convolutions, and concatenates multiple pure transformer module encoders to improve the attention-dependent representation learning performance. Ma et al 31 proposed a homo-heterogenous transformer learning (HHTL) framework for HRRS scene classification according to the characteristics of the transformer to divide the image into multiple patches.…”

Context and Difference Enhancement Network for Change Detection

“…From the available transformation change information, they can distinguish the changed region. In recent years, deep learning methods, especially convolutional neural networks (CNNs), have been widely used in various computer vision (CV) tasks, such as image segmentation [11], [12], [13], scene classification [14], [15], [16], object detection [17], [18], saliency detection [19], [20], image enhancement [21], [22], group detection [23], and so on. Because the CNNs can learn the multi-level features and semantic features of the bi-temporal images, they are also introduced into CD to effectively describe the change information [24], [25], [26].…”