Vision Transformer: An Excellent Teacher for Guiding Small Networks in Remote Sensing Image Scene Classification

Xu, Kejie; Deng, Peifang; Hong, Huixiao

doi:10.1109/tgrs.2022.3152566

Cited by 42 publications

(26 citation statements)

References 86 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The structure of the ViT is completely different from the CNN, which treats the 2D image as the 1D ordered sequence and applies the selfattention mechanism for global dependency modelling, demonstrating stronger global feature extraction. Driven by this, many researchers in the field of remote sensing introduced ViTs for segmentation-related tasks, such as land cover classification [63][64][65][66][67][68], urban scene parsing [69][70][71][72][73][74], change detection [75,76], road extraction [77] and especially building extraction [78]. For example, Chen et al [79] proposed a sparse token Transformer to learn the global dependency of tokens in both spatial and channel dimensions, achieving state-of-the-art accuracy on benchmark building extraction datasets.…”

Section: B Vit-based Building Extraction Methodsmentioning

confidence: 99%

Building Extraction With Vision Transformer

Wang

Fang

et al. 2022

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

As an important carrier of human productive activities, the extraction of buildings is not only essential for urban dynamic monitoring but also necessary for suburban construction inspection. Nowadays, accurate building extraction from remote sensing images remains a challenge due to the complex background and diverse appearances of buildings. The convolutional neural network (CNN) based building extraction methods, although increased the accuracy significantly, are criticized for their inability for modelling global dependencies. Thus, this paper applies the Vision Transformer for building extraction. However, the actual utilization of the Vision Transformer often comes with two limitations. First, the Vision Transformer requires more GPU memory and computational costs compared to CNNs. This limitation is further magnified when encountering large-sized inputs like fine-resolution remote sensing images. Second, spatial details are not sufficiently preserved during the feature extraction of the Vision Transformer, resulting in the inability for fine-grained building segmentation. To handle these issues, we propose a novel Vision Transformer (BuildFormer), with a dual-path structure. Specifically, we design a spatial-detailed context path to encode rich spatial details and a global context path to capture global dependencies. Besides, we develop a window-based linear multi-head self-attention to make the complexity of the multi-head self-attention linear with the window size, which strengthens the global context extraction by using large windows and greatly improves the potential of the Vision Transformer in processing large-sized remote sensing images. The proposed method yields state-of-the-art performance (75.74% IoU) on the Massachusetts building dataset. Code will be available.

show abstract

Section: B Vit-based Building Extraction Methodsmentioning

confidence: 99%

Building Extraction With Vision Transformer

Wang

Fang

et al. 2022

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

show abstract

“…In this framework, the ViT model is used to capture semantic features, while the CNN model is used to extract local structure information. In [47], the advantages of the two models are integrated without improving the computational complexity by knowledge distillation, in which the ViT is worked as a teacher to guide the student model ResNet18. Besides classifying tasks, the paper also proves that this method has good generalization ability for different tasks.…”

Section: Related Workmentioning

confidence: 99%

Vision Transformer With Contrastive Learning for Remote Sensing Image Scene Classification

Wang

et al. 2023

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Remote sensing images (RSIs) are characterized by complex spatial layouts and ground object structures. ViT can be a good choice for scene classification owing to the ability to capture long-range interactive information between patches of input images. However, due to the lack of some inductive biases inherent to CNNs, such as locality and translation equivariance, ViT cannot generalize well when trained on insufficient amounts of data. Compared with training ViT from scratch, transferring a large-scale pre-trained one is more cost-efficient with better performance even when the target data is small-scale. In addition, In addition, the cross-entropy (CE) loss is frequently utilized in scene classification yet has low robustness to noise labels and poor generalization performances for different scenes. In this paper, a ViT-based model in combination with supervised contrastive learning (CL) is proposed, named ViT-CL. For CL, Supervised contrastive (SupCon) loss which is developed by extending the self-supervised contrastive approach to the fullysupervised setting, can explore the label information of RSIs in embedding space and improve the robustness to common image corruption. In ViT-CL, a joint loss function that combines CE loss and SupCon loss is developed to prompt the model to learn more discriminative features. Also, a two-stage optimization framework is introduced to enhance the controllability of the optimization process of the ViT-CL model. Extensive experiments on the AID, NWPU-RESISC45, and UCM datasets verified the superior performance of ViT-CL, with the highest accuracies of 97.42%, 94.54%, and 99.76% among all competing methods, respectively.

show abstract

“…The detailed steps are listed as follows: Given a 3D image, it is first reshaped into a set of non-overlapping and flat 2D patches x n ∈ n×(P 2 •C) , where H, W, C represent the height, width, and channel of the images, P represents the resolution of the patches, and n is the number of embedded patches. Subsequently, the patches are projected to the D dimension by a learnable embedding matrix E ∈ (P 2 •C)×d [56]. Up to this point, an image is transformed into a sequence consisting of n patches which can also be called tokens.…”

Section: Vision Transformermentioning

confidence: 99%

“…Horry et al [55] introduced a highly scalable and accurate ViT model into a low-parameter CNN for land use classification, aiming to cope with the growing amount of satellite data. Similarly, Xu et al [56] proposed an end-to-end network, called ET-GSNet, to improve the generalization of the different tasks by knowledge distillation. Besides, transformer and convolution-based SAR image despeckling networks were proposed by Perera et al [57] that also possessed superior performance compared to the pure CNN.…”

Section: Introductionmentioning

confidence: 99%

Landslide Detection Based on ResU-Net with Transformer and CBAM Embedded: Two Examples with Geologically Different Environments

Xu¹,

2022

Remote Sensing

View full text Add to dashboard Cite

An efficient method of landslide detection can provide basic scientific data for emergency command and landslide susceptibility mapping. Compared to a traditional landslide detection approach, convolutional neural networks (CNN) have been proven to have powerful capabilities in reducing the time consumed for selecting the appropriate features for landslides. Currently, the success of transformers in natural language processing (NLP) demonstrates the strength of self-attention in global semantic information acquisition. How to effectively integrate transformers into CNN, alleviate the limitation of the receptive field, and improve the model generation are hot topics in remote sensing image processing based on deep learning (DL). Inspired by the vision transformer (ViT), this paper first attempts to integrate a transformer into ResU-Net for landslide detection tasks with small datasets, aiming to enhance the network ability in modelling the global context of feature maps and drive the model to recognize landslides with a small dataset. Besides, a spatial and channel attention module was introduced into the decoder to effectually suppress the noise in the feature maps from the convolution and transformer. By selecting two landslide datasets with different geological characteristics, the feasibility of the proposed model was validated. Finally, the standard ResU-Net was chosen as the benchmark to evaluate the proposed model rationality. The results indicated that the proposed model obtained the highest mIoU and F1-score in both datasets, demonstrating that the ResU-Net with a transformer embedded can be used as a robust landslide detection method and thus realize the generation of accurate regional landslide inventory and emergency rescue.

show abstract

Vision Transformer: An Excellent Teacher for Guiding Small Networks in Remote Sensing Image Scene Classification

Cited by 42 publications

References 86 publications

Building Extraction With Vision Transformer

Building Extraction With Vision Transformer

Vision Transformer With Contrastive Learning for Remote Sensing Image Scene Classification

Landslide Detection Based on ResU-Net with Transformer and CBAM Embedded: Two Examples with Geologically Different Environments

Contact Info

Product

Resources

About