Registration of Multisensor Images through a Conditional Generative Adversarial Network and a Correlation-Type Similarity Measure

Abstract

The automatic registration of multisensor remote sensing images is a highly challenging task due to the inherently different physical, statistical, and textural characteristics of the input data. Information-theoretic measures are often used to favor comparing local intensity distributions in the images. In this paper, a novel method based on the combination of a deep learning architecture and a correlation-type area-based functional is proposed for the registration of a multisensor pair of images, including an optical image and a synthetic aperture radar (SAR) image. The method makes use of a conditional generative adversarial network (cGAN) in order to address image-to-image translation across the optical and SAR data sources. Then, once the optical and SAR data are brought to a common domain, an area-based ℓ2 similarity measure is used together with the COBYLA constrained maximization algorithm for registration purposes. While correlation-type functionals are usually ineffective in the application to multisensor registration, exploiting the image-to-image translation capabilities of cGAN architectures allows moving the complexity of the comparison to the domain adaptation step, thus enabling the use of a simple ℓ2 similarity measure, favoring high computational efficiency, and opening the possibility to process a large amount of data at runtime. Experiments with multispectral and panchromatic optical data combined with SAR images suggest the effectiveness of this strategy and the capability of the proposed method to achieve more accurate registration as compared to state-of-the-art approaches.

Registration of Large Optical and SAR Images with Non-Flat Terrain by Investigating Reliable Sparse Correspondences

“…However, we think that the high response locations obtained by these Harris-like operators can only be considered 'salient' in a very limited local neighborhood, about 10 × 10 pixels for the normally applied parameter setting. It does not guarantee the saliency of the local image template, which is usually 100 × 100 to 200 × 200 pixels [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30]. To this end, a deep learning-based feature point detector is proposed in [29], which uses a convolutional network to assess the 'goodness' of the local image patches for template matching.…”

“…Current studies on optical-SAR registration, no matter the handcrafted methods [7][8][9][10][11][12][13][14][15][16][17][18][19] or the deep learning-based ones [20][21][22][23][24][25][26][27][28][29][30], mainly focus on dealing with the vast radiometric and geometric disparity problem, which makes it quite difficult to obtain sufficient reliable CPs that are sparsely distributed across the input image pairs. After the putative CPs are obtained, outlier removal and image warping are mostly conducted under the assumption that the geometric relationship between the input optical-SAR image pairs can be depicted by a linear equation, such as the affine or projective transformation.…”

“…The HOPES method extracts the primary edge structure using the Gabor filters and conducts an edge fusion algorithm to obtain shared features from optical and SAR images. In recent years, many deep learning-based methods have come out [20][21][22][23][24][25][26][27][28][29][30], where diverse kinds of Siamese or pseudo-Siamese convolutional neural network architectures are designed to learn shared features from optical and SAR images. In [27], the CNN feature extractor produces pixelwise deep features, which mimic the handcrafted method.…”

Optical and SAR Image Registration Based on Pseudo-SAR Image Generation Strategy

“…To mitigate the feature differences, several scholars have proposed image transformation-based methods. For instance, Maggiolo et al [25] employed a conditional GAN-based generation strategy to convert optical images into SAR images and then conducted a template matching between the GAN-generated SAR and real SAR images. Huang et al [26] utilized a CycleGAN network structure to transform SAR images into pseudooptical images and registered them with real optical images.…”

Multimodal Satellite Image Time Series Analysis Using GAN-Based Domain Translation and Matrix Profile

“…Recently, inter-modality translation has received considerable attention in the remote sensing community. In this sense, several approaches for transforming images acquired by an EO sensor into images that follow the characteristics of another type of EO sensor have been developed for multimodal SITS analysis [9], bi-temporal change detection [3,10,11], and registration of multi-sensor images [12].…”