Index Your Position: A Novel Self-Supervised Learning Method for Remote Sensing Images Semantic Segmentation

Muhtar, Dilxat; Zhang, Xueliang; Xiao, Pengfeng

doi:10.1109/tgrs.2022.3177770

Cited by 34 publications

(42 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is implemented by forcing the model to learn representations of other related tasks simultaneously, e.g., the coordinates of the input imagery (Muhtar et al, 2022), and the night-time light intensities (Xie et al, 2016). Another is the contrastive learning algorithm, which is to learn the representations by pulling positive (similar) feature pairs closely in latent space and pushing the negative (dissimilar) feature far away from the positive.…”

Section: Weak-supervised Learning Algorithmsmentioning

confidence: 99%

CBRA: The first multi-annual (2016–2021) and high-resolution (2.5 m) building rooftop area dataset in China derived with Super-resolution Segmentation from Sentinel-2 imagery

Liu

Lyu

2023

Preprint

View full text Add to dashboard Cite

Abstract. Large-scale and multi-annual maps of building rooftop area (BRA) are crucial for addressing policy decisions and sustainable development. In addition, as a fine-grained indicator of human activities, BRA could contribute to urban planning and energy modelling to provide benefits to human well-being. However, it is still challenging to produce large-scale BRA due to the rather tiny size of individual buildings. From the viewpoint of classification methods, conventional approaches utilize high-resolution aerial images (metric or sub-metric resolution) to map BRA; unfortunately, high-resolution imagery is both infrequently captured and expensive to purchase, making the BRA mapping costly and inadequate over a consistent spatio-temporal scale. From the viewpoint of learning strategies, there is a non-trivial gap that persists between the limited training references and the applications over geospatial variations. Despite the difficulties, existing large-scale BRA datasets, such as those from Microsoft or Google, do not include China, hence there are no full-coverage maps of BRA in China yet. In this paper, we first propose a deep-learning method, named Spatio-Temporal aware Super-Resolution Segmentation framework (STSR-Seg) to achieve robust super-resolution BRA extraction from relatively low-resolution imagery over a large geographic space. Then, we produce the multi-annual China building rooftop area dataset (CBRA) with 2.5 m resolution from 2016–2021 Sentinel-2 images. The CBRA is the first full-coverage and multi-annual BRA data in China. With the designed training sample generation algorithms and the spatio-temporal aware learning strategies, the CBRA achieves good performance with the F1 score of 62.55 % (+10.61 % compared with the previous BRA data in China) based on 250,000 testing samples in urban areas, and the recall of 78.94 % based on 30,000 testing samples in rural areas. Temporal analysis shows good performance consistency over years and the well agreement to other multi-annual impervious surface area datasets. The STSR-Seg will enable low-cost, dynamic and large-scale BRA mapping (https://github.com/zpl99/STSR-Seg). The CBRA will foster the development of BRA mapping and therefore provide basic data for sustainable research (Liu et al., 2023; https://doi.org/10.5281/zenodo.7500612).

show abstract

Section: Weak-supervised Learning Algorithmsmentioning

confidence: 99%

CBRA: The first multi-annual (2016–2021) and high-resolution (2.5 m) building rooftop area dataset in China derived with Super-resolution Segmentation from Sentinel-2 imagery

Liu

Lyu

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…SSL is a representation learning paradigm that involves training a model to learn from unlabeled data by leveraging pre-text tasks to generate labels or supervision from the data itself [10], [11], [12]. The goal of SSL is to learn general representations that can be used for various downstream tasks, such as classification [42] and semantic segmentation [36]. This section briefly discusses the two dominant learning frameworks of SSL [43]: contrastive learning (CL) and masked image modeling (MIM), and their applications to RS images.…”

Section: Related Workmentioning

confidence: 99%

“…For example, CL was integrated with change detection using multi-modal bitemporal scenes in an encoder-decoder architecture and achieved promising success in multisensor change detection [68]. GLCNet [69] and In-dexNet [36] combined global and local contrast to learn more fine-grained representations for semantic segmentation of RS images.…”

Section: Self-supervised Learning In Remote Sensingmentioning

confidence: 99%

“…2) Adaptation for RS Images: Although the simplicity and effectiveness of SimMIM, there are some limitations must be taken into account when applying SimMIM into RS images. One issue is that SimMIM replaces masked patches with the [MASK] token, but RS images are known for their multiobject characteristics [36] and the objects are usually densely distributed [14]. The masking operation may cause the dense and small objects in the image to be lost [14], leading to incomplete semantic meaning and making image reconstruction more difficult [14].…”

Section: B Masked Image Modeling Branchmentioning

confidence: 99%

See 1 more Smart Citation

CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding

Muhtar

Zhang

Xiao

et al. 2023

IEEE Trans. Geosci. Remote Sensing

Self Cite

View full text Add to dashboard Cite

Self-supervised learning (SSL) has gained widespread attention in the remote sensing (RS) and earth observation (EO) communities owing to its ability to learn task-agnostic representations without human-annotated labels. Nevertheless, most existing RS SSL methods are limited to learning either global semantic separable or local spatial perceptible representations. We argue that this learning strategy is suboptimal in the realm of RS, since the required representations for different RS downstream tasks are often varied and complex. In this study, we proposed a unified SSL framework that is better suited for RS images representation learning. The proposed SSL framework, Contrastive Mask Image Distillation (CMID), is capable of learning representations with both global semantic separability and local spatial perceptibility by combining contrastive learning (CL) with masked image modeling (MIM) in a self-distillation way. Furthermore, our CMID learning framework is architecture-agnostic, which is compatible with both convolutional neural networks (CNN) and vision transformers (ViT), allowing CMID to be easily adapted to a variety of deep learning (DL) applications for RS understanding. Comprehensive experiments have been carried out on four downstream tasks (i.e. scene classification, semantic segmentation, object-detection, and change detection) and the results show that models pre-trained using CMID achieve better performance than other state-of-the-art SSL methods on multiple downstream tasks. The code and pre-trained models will be made available at https://github.com/NJU-LHRS/official-CMID to facilitate SSL research and speed up the development of RS images DL applications.

show abstract

“…Zhang et al [221] introduced a self-supervised spectral-spatial attention-based vision transformer (SSVT), where global and local augmented views are contrasted based on self-distillation [222]. Muhtar et al [223] proposed In-dexNet, a dense self-supervised method for remote sensing image segmentation. Their approach is built on BYOL and performs contrastive at image and pixel level to preserve spatial information.…”

Section: Infoncementioning

confidence: 99%

Self-Supervised Learning in Remote Sensing: A review

Wang

Albrecht

Braham

et al. 2022

IEEE Geosci. Remote Sens. Mag.

102

View full text Add to dashboard Cite

In deep learning research, self-supervised learning (SSL) has received great attention triggering interest within both the computer vision and remote sensing communities. While there has been a big success in computer vision, most of the potential of SSL in the domain of earth observation remains locked. In this paper, we provide an introduction to, and a review of the concepts and latest developments in SSL for computer vision in the context of remote sensing. Further, we provide a preliminary benchmark of modern SSL algorithms on popular remote sensing datasets, verifying the potential of SSL in remote sensing and providing an extended study on data augmentations. Finally, we identify a list of promising directions of future research in SSL for earth observation (SSL4EO) to pave the way for fruitful interaction of both domains.

show abstract

Index Your Position: A Novel Self-Supervised Learning Method for Remote Sensing Images Semantic Segmentation

Cited by 34 publications

References 41 publications

CBRA: The first multi-annual (2016–2021) and high-resolution (2.5 m) building rooftop area dataset in China derived with Super-resolution Segmentation from Sentinel-2 imagery

CBRA: The first multi-annual (2016–2021) and high-resolution (2.5 m) building rooftop area dataset in China derived with Super-resolution Segmentation from Sentinel-2 imagery

CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding

Self-Supervised Learning in Remote Sensing: A review

Contact Info

Product

Resources

About