A Deep Multi-Attention Driven Approach for Multi-Label Remote Sensing Image Classification

Abstract

Deep learning (DL) based methods have been found popular in the framework of remote sensing (RS) image scene classification. Most of the existing DL based methods assume that training images are annotated by single-labels, however RS images typically contain multiple classes and thus can simultaneously be associated with multi-labels. Despite the success of existing methods in describing the information content of very high resolution aerial images with RGB bands, any direct adaptation for highdimensional high-spatial resolution RS images falls short of accurate modeling the spectral and spatial information content. To address this problem, this paper presents a novel approach in the framework of the multi-label classification of high dimensional RS images. The proposed approach is based on three main steps. The first step describes the complex spatial and spectral content of image local areas by a novel K-Branch CNN that includes spatial resolution specific CNN branches. The second step initially characterizes the importance scores of different local areas of each image and then defines a global descriptor for each image based on these scores. This is achieved by a novel multi-attention strategy that utilizes the bidirectional long short-term memory networks. The final step achieves the classification of RS image scenes with multi-labels. Experiments carried out on BigEarthNet (which is a large-scale Sentinel-2 benchmark archive) show the effectiveness of the proposed approach in terms of multi-label classification accuracy compared to the state-of-the-art approaches. The code of the proposed approach is publicly available at https://gitlab.tubit.tu-berlin.de/rsim/MAML-RSIC.

Bigearthnet: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding

“…For example, class Airports essentially relates to land use, and Intertidal flats appear in RS images either with or without water depending on the image acquisition time and hence require appropriate time series for its classification. The number of labels associated with each image pair varies between 1 and 12, while 96.80% of image 3 https://bigearth.eu/BigEarthNetListofClasses.pdf pairs are not associated with more than 5 labels. Only 23 image pairs are annotated with more than 9 labels.…”

MultiScene: A Large-Scale Dataset and Benchmark for Multiscene Recognition in Single Aerial Images

“…Hua et al [54] proposed an endto-end network comprising a CNN and a long short-term memory (LSTM) network that is responsible for modeling label dependencies through its recurrent units for multilabel object classification. Sumbul and Demir [55] exploited a bidirectional LSTM network to learn spatial relations among all patches in an image for the final prediction. Hua et al [56] proposed a relational reasoning network module to model label dependencies and gained better classification results.…”

UAV Image Multi-Labeling with Data-Efficient Transformers

“…Hua et al [20] proposed an end-to-end architecture that consists of three modules, one for learning high-level feature from the high-resolution aerial image, the second is an attention layer to keep only features located in the discriminative regions, and the last module is a bidirectional LSTM for utilizing the relations among labels in both directions to produce the final labels. In [17], the authors proposed a method for multi-label classification for high-dimensional varying spatial resolutions remote sensing imagery. First, a multi-branch CNN is used to describe the local area of image bands with a branch dedicated to each spatial resolution.…”

“…On the contrary, annotation during scene classification requires labels at the scene level only. For these reasons, the interest in multi-label classification, which aims to assign an image with multiple semantic labels, is increasing in the community [5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20]. It is an essential step to provide a better understanding of the scene.…”