Improved SmapGAN remote sensing image map generation based on multi-head self-attention and carafe

Abstract

The changes in ground roads, buildings, and occurrences of natural disasters lead to mismatches between the actual ground conditions and existing maps. Through style transfer between real-time remote sensing images and maps, map content can be rapidly generated and updated. However, in existing methods for generating maps from remote sensing images based on SmapGAN, we first found that using ResBlock as the style conversion module fails to establish long-distance relationships between features. In addition, the small receptive field of convolution layers in ResBlock leads to poor global information capture, resulting in inferior image restoration during upsampling. Second, using transpose convolution as the upsampling method can result in the issue of blurred content in the generated maps. To address these problems, we propose corresponding improvements: on one hand, a style conversion module combining multi-headed self-attention (MHSA) with residual modules, named MHSA-ResBlock, is introduced to address the difficulty in capturing long-distance relationships between features when dealing with a large number of pixel features, and to better capture global information in images. On the other hand, an upsampling method combining transpose convolution with the CARAFE upsampling operator, named TC-Carafe, is proposed to tackle the issues of content loss and blurring associated with traditional transpose convolution upsampling. Furthermore, experimental results show that the MHSA-ResBlock establishes inter-pixel feature relationships and leverages the advantages of fine-grained upsampling operations with TC-Carafe, thereby utilizing inter-pixel feature relationships and neighborhood information to further improve the quality of map generation. Compared to SmapGAN, our research method has shown improvements of 0.6133 and 0.0042 in PSNR and SSIM, respectively. In addition, it has reduced RMSE by 0.72, outperforming SmapGAN in all metrics.