Retrieval Topic Recurrent Memory Network for Remote Sensing Image Captioning

Abstract

Remote sensing image (RSI) captioning aims to generate sentences to describe the content of RSIs. Generally, five sentences are used to describe the RSI in caption datasets. Every sentence can just focus on part of images' contents due to the different attention parts of annotation persons. One annotated sentence may be ambiguous compared with other four sentences. However, previous methods, treating five sentences separately, may generate an ambiguous sentence. In order to consider five sentences together, a collection of words, which named topic words contained common information among five sentences, is jointly incorporated into a captioning model to generate a determinate sentence that covers common contents in RSIs. Instead of employing a naive recurrent neural network, a memory network in which topic words can be naturally included as memory cells is introduced to generate sentences. A novel retrieval topic recurrent memory network is proposed to utilize the topic words. First, a topic repository is built to record the topic words in training datasets. Then, the retrieval strategy is exploited to obtain the topic words for a test image from topic repository. Finally, the retrieved topic words are incorporated into a recurrent memory network to guide the sentence generation. In addition to getting topics through retrieval, the topic words of test images can also be edited manually. The proposed method sheds light on controllability of caption generation. Experiments are conducted on two caption datasets to evaluate the proposed method.

Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information

“…To solve the problem that the current caption generator requires high computing power, Genc et al proposed a decoder based on support vector machine [26], which is effective when just a limited amount of training samples is available. Despite the relative maturity of caption-based RSCTIR [12], [27]- [29], the model inevitably introduces noise due to the limitations of dual-stage process, thus affecting the accuracy of retrieval.…”

Using Neural Encoder-Decoder Models With Continuous Outputs for Remote Sensing Image Captioning

“…Besides tailored attention mechanisms, other previous studies on remote sensing image captioning have explored alternative paths for improving the results of standard encoderdecoder neural models. These include studies (a) exploring multi-scale feature representations [15], [16]; (b) using novel loss functions [17] or training procedures based on reinforcement learning [18], improving on the standard cross-entropy loss [17]; (c) extending and combining the set of reference captions, associated to each image, through summarization [19] or retrieval [20] approaches; or (d) using decoder components based on the Transformer architecture [18].…”

A Confounder-Free Fusion Network for Aerial Image Scene Feature Representation

“…Aerial image captioning: Aerial image captioning aims to accurately describe the image by generating concise and flexible sentences. Recently, many aerial image captioning methods [34][35][36][37][38] have been proposed for a better understanding of image content. They are aided by three public datasets, UCM-captions dataset [39], Sydney-captions dataset [39], and RSICD [40].…”

“…A sound active attention framework is proposed in [34] to generate image descriptions with the guidance of sound. Furthermore, based on the existing five sentences in datasets, a topic word strategy is proposed in [35] to describe image with a memory network. In addition, a visual aligning attention model is proposed in [17] to provide accurate image captioning by focusing on regions of interest.…”