Description Generation for Remote Sensing Images Using Attribute Attention Mechanism

Zhang, Xiangrong; Wang, Xin; Tang, Xu; Zhou, Huiyu; Li, Chen

doi:10.3390/rs11060612

Cited by 91 publications

(55 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, Wang et al [8] proposed a novel method which used Latent semantic embedding by metric learning for remote sensing image multi-sentence captioning task. Zhang et al [9] propose a new model with an attribute attention mechanism for remote sensing image captioning. At the same time, it is explored whether the attributes extracted from remote sensing images have an impact on the attention mechanism.…”

Section: Remote Sensing Image Captioningmentioning

confidence: 99%

“…In remote sensing image captioning [4][5][6][7][8][9], the above principle shows that the accurate recognition of remote sensing images often requires the spatial relationships between objects. Especially in the cases of different objects with the same spectrum, different types of remote sensing object may have the same spectral, texture and shape features and can only be more accurately interpreted by using adjacent objects and their spatial relations.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Scale Remote Sensing Semantic Analysis Based on a Global Perspective

Cui

Zhang

et al. 2019

IJGI

View full text Add to dashboard Cite

Remote sensing image captioning involves remote sensing objects and their spatial relationships. However, it is still difficult to determine the spatial extent of a remote sensing object and the size of a sample patch. If the patch size is too large, it will include too many remote sensing objects and their complex spatial relationships. This will increase the computational burden of the image captioning network and reduce its precision. If the patch size is too small, it often fails to provide enough environmental and contextual information, which makes the remote sensing object difficult to describe. To address this problem, we propose a multi-scale semantic long short-term memory network (MS-LSTM). The remote sensing images are paired into image patches with different spatial scales. First, the large-scale patches have larger sizes. We use a Visual Geometry Group (VGG) network to extract the features from the large-scale patches and input them into the improved MS-LSTM network as the semantic information, which provides a larger receptive field and more contextual semantic information for small-scale image caption so as to play the role of global perspective, thereby enabling the accurate identification of small-scale samples with the same features. Second, a small-scale patch is used to highlight remote sensing objects and simplify their spatial relations. In addition, the multi-receptive field provides perspectives from local to global. The experimental results demonstrated that compared with the original long short-term memory network (LSTM), the MS-LSTM’s Bilingual Evaluation Understudy (BLEU) has been increased by 5.6% to 0.859, thereby reflecting that the MS-LSTM has a more comprehensive receptive field, which provides more abundant semantic information and enhances the remote sensing image captions.

show abstract

Section: Remote Sensing Image Captioningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multi-Scale Remote Sensing Semantic Analysis Based on a Global Perspective

Cui

Zhang

et al. 2019

IJGI

View full text Add to dashboard Cite

show abstract

“…This detailed information is helpful for generating detailed description of objects or relationships in images. Later on, X. Zhang et al [19] added an attention mechanism into the remote sensing image caption models. In addition, the high-level image features were also used as the attributes.…”

Section: Introductionmentioning

confidence: 99%

LAM: Remote Sensing Image Captioning with Label-Attention Mechanism

et al. 2019

View full text Add to dashboard Cite

Significant progress has been made in remote sensing image captioning by encoder-decoder frameworks. The conventional attention mechanism is prevalent in this task but still has some drawbacks. The conventional attention mechanism only uses visual information about the remote sensing images without considering using the label information to guide the calculation of attention masks. To this end, a novel attention mechanism, namely Label-Attention Mechanism (LAM), is proposed in this paper. LAM additionally utilizes the label information of high-resolution remote sensing images to generate natural sentences to describe the given images. It is worth noting that, instead of high-level image features, the predicted categories’ word embedding vectors are adopted to guide the calculation of attention masks. Representing the content of images in the form of word embedding vectors can filter out redundant image features. In addition, it can also preserve pure and useful information for generating complete sentences. The experimental results from UCM-Captions, Sydney-Captions and RSICD demonstrate that LAM can improve the model’s performance for describing high-resolution remote sensing images and obtain better S m scores compared with other methods. S m score is a hybrid scoring method derived from the AI Challenge 2017 scoring method. In addition, the validity of LAM is verified by the experiment of using true labels.

show abstract

“…The bidirectional grid LSTM took visual features of an image as the input and learned complex spatial patterns based on two-dimensional context.In recent years, the application of reinforcement learning [20][21][22][23] in image caption has also been a hot topic, which adjusts the generation strategies using the change of the reward functions in the caption generation process to dynamic vocabulary generation.However, most of the current studies focus on the scene semantic description of ordinary digital images [24,25]. To use the deep RNN or LSTM to execute the semantic analysis [26][27][28][29][30] of remote sensing objects, the following problems must be solved:Location ambiguity: At different time steps, the attention mechanism is based on 14 × 14-sized image features and corresponds to 196 spatial locations in remote sensing images. There are some deviations [31], however, that limit the application in remote sensing object recognition.Boundary ambiguity: the nouns (label of objects) in captions cannot accurately segment the boundaries of remote sensing objects in an image; thus, it is impossible to identify the spatial relationship between the objects.…”

mentioning

confidence: 99%

“…However, most of the current studies focus on the scene semantic description of ordinary digital images [24,25]. To use the deep RNN or LSTM to execute the semantic analysis [26][27][28][29][30] of remote sensing objects, the following problems must be solved:…”

mentioning

confidence: 99%

Multi-Scale Semantic Segmentation and Spatial Relationship Recognition of Remote Sensing Images Based on an Attention Model

Cui

Wang

et al. 2019

Remote Sensing

View full text Add to dashboard Cite

A comprehensive interpretation of remote sensing images involves not only remote sensing object recognition but also the recognition of spatial relations between objects. Especially in the case of different objects with the same spectrum, the spatial relationship can help interpret remote sensing objects more accurately. Compared with traditional remote sensing object recognition methods, deep learning has the advantages of high accuracy and strong generalizability regarding scene classification and semantic segmentation. However, it is difficult to simultaneously recognize remote sensing objects and their spatial relationship from end-to-end only relying on present deep learning networks. To address this problem, we propose a multi-scale remote sensing image interpretation network, called the MSRIN. The architecture of the MSRIN is a parallel deep neural network based on a fully convolutional network (FCN), a U-Net, and a long short-term memory network (LSTM). The MSRIN recognizes remote sensing objects and their spatial relationship through three processes. First, the MSRIN defines a multi-scale remote sensing image caption strategy and simultaneously segments the same image using the FCN and U-Net on different spatial scales so that a two-scale hierarchy is formed. The output of the FCN and U-Net are masked to obtain the location and boundaries of remote sensing objects. Second, using an attention-based LSTM, the remote sensing image captions include the remote sensing objects (nouns) and their spatial relationships described with natural language. Finally, we designed a remote sensing object recognition and correction mechanism to build the relationship between nouns in captions and object mask graphs using an attention weight matrix to transfer the spatial relationship from captions to objects mask graphs. In other words, the MSRIN simultaneously realizes the semantic segmentation of the remote sensing objects and their spatial relationship identification end-to-end. Experimental results demonstrated that the matching rate between samples and the mask graph increased by 67.37 percentage points, and the matching rate between nouns and the mask graph increased by 41.78 percentage points compared to before correction. The proposed MSRIN has achieved remarkable results. the existing studies do not address the interpretation of spatial relationships between remote sensing objects, which limits the understanding of remote sensing objects, especially when the phenomenon of different objects with the same spectrum in remote sensing appears.The phenomenon of different objects with the same spectrum in remote sensing is quite common. It is difficult to identify objects only by their own textures, spectra, and shape information. Object identification requires multi-scale semantic information and spatially adjacent objects to assist in decision-making. The spatial relationship between remote sensing objects is of great significance to the recognition of remote sensing objects when different objects have the same spectru...

show abstract

Description Generation for Remote Sensing Images Using Attribute Attention Mechanism

Cited by 91 publications

References 30 publications

Multi-Scale Remote Sensing Semantic Analysis Based on a Global Perspective

Multi-Scale Remote Sensing Semantic Analysis Based on a Global Perspective

LAM: Remote Sensing Image Captioning with Label-Attention Mechanism

Multi-Scale Semantic Segmentation and Spatial Relationship Recognition of Remote Sensing Images Based on an Attention Model

Contact Info

Product

Resources

About