MAPMaN: Multi‐Stage U‐Shaped Adaptive Pattern Matching Network for Semantic Segmentation of Remote Sensing Images

Abstract

Remote sensing images (RSIs) often possess obvious background noises, exhibit a multi‐scale phenomenon, and are characterized by complex scenes with ground objects in diversely spatial distribution pattern, bringing challenges to the corresponding semantic segmentation. CNN‐based methods can hardly address the diverse spatial distributions of ground objects, especially their compositional relationships, while Vision Transformers (ViTs) introduce background noises and have a quadratic time complexity due to dense global matrix multiplications. In this paper, we introduce Adaptive Pattern Matching (APM), a lightweight method for long‐range adaptive weight aggregation. Our APM obtains a set of pixels belonging to the same spatial distribution pattern of each pixel, and calculates the adaptive weights according to their compositional relationships. In addition, we design a tiny U‐shaped network using the APM as a module to address the large variance of scales of ground objects in RSIs. This network is embedded after each stage in a backbone network to establish a Multi‐stage U‐shaped Adaptive Pattern Matching Network (MAPMaN), for nested multi‐scale modeling of ground objects towards semantic segmentation of RSIs. Experiments on three datasets demonstrate that our MAPMaN can outperform the state‐of‐the‐art methods in common metrics. The code can be available at https://github.com/INiid/MAPMaN.