A Two-Stream Symmetric Network with Bidirectional Ensemble for Aerial Image Matching

Abstract

In this paper, we propose a novel method to precisely match two aerial images that were obtained in different environments via a two-stream deep network. By internally augmenting the target image, the network considers the two-stream with the three input images and reflects the additional augmented pair in the training. As a result, the training process of the deep network is regularized and the network becomes robust for the variance of aerial images. Furthermore, we introduce an ensemble method that is based on the bidirectional network, which is motivated by the isomorphic nature of the geometric transformation. We obtain two global transformation parameters without any additional network or parameters, which alleviate asymmetric matching results and enable significant improvement in performance by fusing two outcomes. For the experiment, we adopt aerial images from Google Earth and the International Society for Photogrammetry and Remote Sensing (ISPRS). To quantitatively assess our result, we apply the probability of correct keypoints (PCK) metric, which measures the degree of matching. The qualitative and quantitative results show the sizable gap of performance compared to the conventional methods for matching the aerial images. All code and our trained model, as well as the dataset are available online.Remote Sens. 2020, xx, 5 2 of 21 these approaches are not very successful for aerial images due to their high-resolution, computational costs, large-scale transformation, and variation in the environments.

A Two-Stage Deep Learning Registration Method for Remote Sensing Images Based on Sub-Image Matching

“…Feature maps can be regarded as d-dimensional dense local feature descriptors from the input images. We adopted the correlation layer used in the regression task in [45] as feature correlation here, as the way it computes similarities in the regression can be extended to the matching task. Feature correlation outputs correlation maps C AB ∈ R (h×w)×h×w made up of scalar products of feature descriptors v ∈ R d at each position in a pair of feature maps.…”

“…SE-ResNeXt, which applies the Squeeze-and-Excitation (SE) module to ResNeXt, focuses on more important features. We embed the first three layers of the backbone and apply L2-normalization [45]. Given the coordinates (x s , y s ) and (x t , y t ) corresponding to the sub-image pair (I s , I t ) at the upper left corner or center of the original images, we can obtain the 2-D vectors v A and v B by position embedding, as follows:…”

“…We adopt the grid loss [45] for training. Given the estimation result,Â, and ground truth, A GT , of the global transformation, the loss function is defined as follows:…”

“…The dataset we mainly used included multi-temporal remote sensing image pairs with correct geographical correspondence collected by [45], called the Google Earth (GE) dataset in this paper, as well as some of the images released by the International Society for Photogrammetry and Remote Sensing (ISPRS) [51] and Wuhan University (WHU) Building Dataset [52] for further test. The images in the GE dataset were from Google Earth and were taken at different times (2019, 2017, and 2015) and with different sensors (Landsat-7, Landsat-8, Worldview, and Quickbird).…”

Computational Intelligence in Remote Sensing Image Registration: A survey

“…Experimental results show that this method has a high accuracy. Vakalopoulou et al [122] proposed a CNN-based network for estimating the rigid and deformable parameters between high-resolution satellite image pairs, and then used a 2D space converter layer to align the source image with the target image. Park et al [123] proposed an end-to-end scalable network with a two-stream architecture and a bidirectional training architecture for estimating the transformation parameters for aerial image pairs.…”