A Semantic Guidance and Transformer-Based Matching Method for UAVs and Satellite Images for UAV Geo-Localization

Abstract

It is a challenging task for unmanned aerial vehicles (UAVs) without a positioning system to locate targets by using images. Matching drone and satellite images is one of the key steps in this task. Due to the large angle and scale gap between drone and satellite views, it is very important to extract finegrained features with strong characterization ability. Most of the published methods are based on the CNN structure, but a lot of information will be lost when using such methods. This is caused by the limitations of the convolution operation (e.g. limited receptive field and downsampling operation). To make up for this shortcoming, a transformer-based network is proposed to extract more contextual information. The network promotes feature alignment through semantic guidance module (SGM). SGM aligns the same semantic parts in the two images by classifying each pixel in the images based on the attention of pixels. In addition, this method can be easily combined with existing methods. The proposed method has been implemented with the newest UAV-based geo-localization dataset. Compared with the existing state-of-the-art (SOTA) method, the proposed method achieves almost 8% improvement in accuracy. INDEX TERMSCross-view image matching, geo-localization, UAV image localization, deep neural network.

A Satellite-Drone Image Cross-View Geolocalization Method Based on Multi-Scale Information and Dual-Channel Attention Mechanism

“…In the Satellite->Drone task, R@1 is 92.30% and AP is 87.66%. Its performance surpasses advanced methods such as LPN [29], FSRA [4], SGM [32], and PAAN [50]. On the self-made dataset, the accuracy exhibits a trend similar to that on the University-1652 dataset: our method outperforms the aforementioned two methods.…”

“…Dai [4] achieved automatic region segmentation based on the heat distribution of Transformer feature maps, aligning specific regions in different views to improve the model's accuracy and robustness to location variations. Zhuang [32] proposed a Transformer-based network to match drone images with satellite images. This network classifies each pixel in the image using pixel-wise attention, matching the same semantic parts in two images.…”

“…Although extracting robust global features at multiple scales is crucial, previous research has highlighted the importance of part-based methods for image retrieval [4,29,32]. Aligning specific semantic parts in images is a simple way to implement an end-to-end trained part-based approach.…”

A Transformer and Visual Foundation Model-Based Method for Cross-View Remote Sensing Image Retrieval

“…For instance, the FSRA (Dai et al, 2021) automatically divided the original image into multiple regions based on the heat distribution of the feature map, and achieved feature alignment based on region consistency. Zhuang et al (2022) introduced semantic constraints based on FSRA to enhance the effectiveness of feature alignment. However, a limitation shared by all the above methods is that the training process still involves the backbone network, resulting in increased computational and time overhead for the retrieval task.…”

GeoFormer: An Effective Transformer-Based Siamese Network for UAV Geolocalization

“…In contrast, Swin Transformer [22] introduces a hierarchical structure and a shift window mechanism, which enhances the model's ability to capture spatial relationships between neighboring regions through window sliding and offsetting. As a result, Swin Transformer is able to capture local features in an image more effectively, while ensuring the integration of global information through cross-window connections [60]. In addition, Swin Transformer improves computational efficiency by performing self-attention computation independently within each window, enabling the model to process windows in parallel.…”