Multi-Resolution Transformer Network for Building and Road Segmentation of Remote Sensing Image

Sun, Zhongyu; Zhou, Wei; Ding, Chen; Xia, Min

doi:10.3390/ijgi11030165

Cited by 43 publications

(24 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The structure of the ViT is completely different from the CNN, which treats the 2D image as the 1D ordered sequence and applies the selfattention mechanism for global dependency modelling, demonstrating stronger global feature extraction. Driven by this, many researchers in the field of remote sensing introduced ViTs for segmentation-related tasks, such as land cover classification [63][64][65][66][67][68], urban scene parsing [69][70][71][72][73][74], change detection [75,76], road extraction [77] and especially building extraction [78]. For example, Chen et al [79] proposed a sparse token Transformer to learn the global dependency of tokens in both spatial and channel dimensions, achieving state-of-the-art accuracy on benchmark building extraction datasets.…”

Section: B Vit-based Building Extraction Methodsmentioning

confidence: 99%

Building Extraction With Vision Transformer

Wang

Fang

et al. 2022

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

As an important carrier of human productive activities, the extraction of buildings is not only essential for urban dynamic monitoring but also necessary for suburban construction inspection. Nowadays, accurate building extraction from remote sensing images remains a challenge due to the complex background and diverse appearances of buildings. The convolutional neural network (CNN) based building extraction methods, although increased the accuracy significantly, are criticized for their inability for modelling global dependencies. Thus, this paper applies the Vision Transformer for building extraction. However, the actual utilization of the Vision Transformer often comes with two limitations. First, the Vision Transformer requires more GPU memory and computational costs compared to CNNs. This limitation is further magnified when encountering large-sized inputs like fine-resolution remote sensing images. Second, spatial details are not sufficiently preserved during the feature extraction of the Vision Transformer, resulting in the inability for fine-grained building segmentation. To handle these issues, we propose a novel Vision Transformer (BuildFormer), with a dual-path structure. Specifically, we design a spatial-detailed context path to encode rich spatial details and a global context path to capture global dependencies. Besides, we develop a window-based linear multi-head self-attention to make the complexity of the multi-head self-attention linear with the window size, which strengthens the global context extraction by using large windows and greatly improves the potential of the Vision Transformer in processing large-sized remote sensing images. The proposed method yields state-of-the-art performance (75.74% IoU) on the Massachusetts building dataset. Code will be available.

show abstract

Section: B Vit-based Building Extraction Methodsmentioning

confidence: 99%

Building Extraction With Vision Transformer

Wang

Fang

et al. 2022

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

show abstract

“…Tao et al [26] designed a spatial information inference structure (SIIS-Net), enabling multidirectional message passing across the rows and columns of feature maps to guess occluded roads. Transformer-based methods [27]- [30] calculate the relationships between all pixels on a feature map using a self-attention module. In a self-attention module, each pixel has a global receptive field, thus supporting a stronger global-contextual structure reasoning ability than the conventional spatial attention modules found in CoANet and SIIS-Net, greatly improving the correctness and the topological completeness in road extraction results.…”

Section: A Road Extractionmentioning

confidence: 99%

“…The scale parameter is a coefficient number that controls the number of the dominant queries in our proposed Dominant-Transformer block. We set the value of the scale parameter to [1,3,5,7,9,10,30,50,90] and tested their performance. The evaluation indicators of different scale values are shown in TABLE V. The third column in the table is the value of U , the number of dominant queries.…”

Section: Effect Of the Scale Parameter Of Dominant Transformermentioning

confidence: 99%

UGRoadUpd: An Unchanged-Guided Historical Road Database Updating Framework Based on Bi-Temporal Remote Sensing Images

Zhou

Sui

Chen

et al. 2022

IEEE Trans. Intell. Transport. Syst.

View full text Add to dashboard Cite

Timely updated road networks are the basis for many real-world applications such as intelligent navigation and traffic management. Existing road updating methods based on remote sensing images learn from historical road databases to update roads. Road extraction models learned from historical images however, are not easily applied to a current image due to spectral differences; and only changed roads need updating. In this paper, an Unchanged-Guided Road Updating (UGRoad-Upd) framework is proposed to improve the quality of updated road networks by limiting the road updating range and learning from historical unchanged roads. The UGRoadUpd framework identifies road changes using a novel dual-task dominanttransformer-based neural network for road change detection (DT-RoadCDNet). DT-RoadCDNet executes road segmentation and change detection simultaneously, from bi-temporal remote sensing images. The Dominant-Transformer based Global Context Modeling module in DT-RoadCDNet globally models the contextual spatial structure for improved integrity in roads and road changes. Based on the discovery of road changes, an unchangedguided road update strategy updates the roads in changed areas by learning from the prior information provided by unchanged roads in a historical road database. Experiments on two newly annotated road change detection and update datasets confirms the effectiveness of our UGRoadUpd framework.

show abstract

“…Transformers in segmentation. Very recently transformers have started to be used in that segmentation of overhead imagery [17,42], achieving good performance. Transformers had already started to become common in other domains such as TransUnet [7], ViT-V-Net [6], TransClaw U-Net [5], UTNet [13], Cotr [45], and SwinUnet [3] in medical image segmentation.…”

Section: Related Workmentioning

confidence: 99%

Transformers For Recognition In Overhead Imagery: A Reality Check

Luzi¹,

Gupta²,

Collins³

et al. 2022

Preprint

View full text Add to dashboard Cite

There is evidence that transformers offer state-of-theart recognition performance on tasks involving overhead imagery (e.g., satellite imagery). However, it is difficult to make unbiased empirical comparisons between competing deep learning models, making it unclear whether, and to what extent, transformer-based models are beneficial. In this paper we systematically compare the impact of adding transformer structures into state-of-the-art segmentation models for overhead imagery. Each model is given a similar budget of free parameters, and their hyperparameters are optimized using Bayesian Optimization with a fixed quantity of data and computation time. We conduct our experiments with a large and diverse dataset comprising two large public benchmarks: Inria and DeepGlobe. We perform additional ablation studies to explore the impact of specific transformer-based modeling choices. Our results suggest that transformers provide consistent, but modest, performance improvements. We only observe this advantage however in hybrid models that combine convolutional and transformer-based structures, while fully transformerbased models achieve relatively poor performance.

show abstract

Multi-Resolution Transformer Network for Building and Road Segmentation of Remote Sensing Image

Cited by 43 publications

References 30 publications

Building Extraction With Vision Transformer

Building Extraction With Vision Transformer

UGRoadUpd: An Unchanged-Guided Historical Road Database Updating Framework Based on Bi-Temporal Remote Sensing Images

Transformers For Recognition In Overhead Imagery: A Reality Check

Contact Info

Product

Resources

About