Exploring the Potential of Conditional Adversarial Networks for Optical and SAR Image Matching

Merkle, Nina; Auer, Stefan; Müller, Rupert; Reinartz, Peter

doi:10.1109/jstars.2018.2803212

Cited by 129 publications

(78 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experimental results showed a good performance of the network even without a priori known features. In a similar manner, [5] demonstrated how this form of image-to-image translation can help to ease the coregistration of SAR and optical images for an improvement of the geolocation of optical data. Although the training was experienced to be quite expensive, and although it remains difficult to evaluate artificial, GAN-generated images by standard metrics, visual inspection and comparison to standard similarity measures such as NCC, BRISK and SIFT show the great potential of this approach.…”

Section: Related Workmentioning

confidence: 91%

“…This paper addresses one possible destination point, using an adapted conditional generative adversarial network, taking domain-specific potentials and peculiarities into specific consideration. The discipline of conditional adversarial networks (cGANs) offers promising strategies for generating artificial images and has already successfully been adapted to tasks in multi-sensor remote sensing, e.g., [5][6][7]. Considering the current state-of-the-art, this paper investigates the utilization of an adapted version of the CycleGAN architecture [8] for the SAR-to-optical image translation task and the value of domain knowledge (e.g., initialization, sensor/image characteristics) for improving results.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SAR-to-Optical Image Translation Based on Conditional Generative Adversarial Networks—Optimization, Opportunities and Limits

et al. 2019

Self Cite

View full text Add to dashboard Cite

Due to its all time capability, synthetic aperture radar (SAR) remote sensing plays an important role in Earth observation. The ability to interpret the data is limited, even for experts, as the human eye is not familiar to the impact of distance-dependent imaging, signal intensities detected in the radar spectrum as well as image characteristics related to speckle or steps of post-processing. This paper is concerned with machine learning for SAR-to-optical image-to-image translation in order to support the interpretation and analysis of original data. A conditional adversarial network is adopted and optimized in order to generate alternative SAR image representations based on the combination of SAR images (starting point) and optical images (reference) for training. Following this strategy, the focus is set on the value of empirical knowledge for initialization, the impact of results on follow-up applications, and the discussion of opportunities/drawbacks related to this application of deep learning. Case study results are shown for high resolution (SAR: TerraSAR-X, optical: ALOS PRISM) and low resolution (Sentinel-1 and -2) data. The properties of the alternative image representation are evaluated based on feedback from experts in SAR remote sensing and the impact on road extraction as an example for follow-up applications. The results provide the basis to explain fundamental limitations affecting the SAR-to-optical image translation idea but also indicate benefits from alternative SAR image representations.

show abstract

Section: Related Workmentioning

confidence: 91%

Section: Introductionmentioning

confidence: 99%

SAR-to-Optical Image Translation Based on Conditional Generative Adversarial Networks—Optimization, Opportunities and Limits

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Taking a different approach to the problem, (Merkle et al, 2018) proposed the use of a generative adversarial network (GAN) to generate SAR-like templates from optical image patches. These templates were then used as input to standard template matching approaches such as mutual information (MI) or normalized cross correlation (NCC).…”

Section: Deep Learning For Sar-optical Matchingmentioning

confidence: 99%

A Semi-Supervised Approach to Sar-Optical Image Matching

Hughes

Schmitt

2019

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

<p><strong>Abstract.</strong> Matching synthetic aperture radar (SAR) and optical remote sensing imagery is a key first step towards exploiting the complementary nature of these data in data fusion frameworks. While numerous signal-based approaches to matching have been proposed, they often fail to perform well in multi-sensor situations. In recent years deep learning has become the go-to approach for solving image matching in computer vision applications, and has also been adapted to the case of SAR-optical image matching. However, the hitherto proposed techniques still fail to match SAR and optical imagery in a generalizable manner. These limitations are largely due to the complexities in creating large-scale datasets of corresponding SAR and optical image patches. In this paper we frame the matching problem within semi-supervised learning, and use this as a proxy for investigating the effects of data scarcity on matching. In doing so we make an initial contribution towards the use of semi-supervised learning for matching SAR and optical imagery. We further gain insight into the non-complementary nature of commonly used supervised and unsupervised loss functions, as well as dataset size requirements for semi-supervised matching.</p>

show abstract

“…The coupled optical and SAR patches for different sources are then automatically extracted by the learning features in the pretrained network Pseudo-Siamese CNN [47] and generative matching network (GMN) [48], respectively. At the same time, in [49], the corresponding SAR-like images are constructed via conditional generative adversarial networks (cGANs). In this regard, improving the accuracy of ground control points selection proves to be effective.…”

Section: Training Algorithms In B_cnn and Trans_cnnmentioning

confidence: 99%

Transfer and Association: A Novel Detection Method for Targets without Prior Homogeneous Samples

Zhou

Zhang

2019

Remote Sensing

View full text Add to dashboard Cite

A primary problem faced during previous research was the gap in limited and unbalanced quantity of prior samples between computer classification tasks and targeted remote sensing applications. This paper presents the fusion method to overcome this limitation. It offers a novel method based on knowledge transfer and feature association, a strong combination of transfer learning and data fusion. The former reuses layers trained on complete data sets to compute a mid-level representation of the specific target. The latter brings additional information from heterogeneous sources to enrich the features in the target domain. Firstly, a basic convolutional neural network (B_CNN) is pretrained on to the CIFAR-10 dataset to produce a stable model responsible for general feature extraction from multiple inputs. Secondly, a transfer CNN (Trans_CNN) with fine-tuned and transferred parameters is constraint-trained to fit and switch between differing tasks. Meanwhile, the feature association (FA) frames a new feature space to achieve integration between training and testing samples from different sensors. Finally, on-line detection can be completed based on Trans_CNN to explore a state-of-the-art method to overcome the inadequate sample problems in real remote sensing applications rather than produce an unrolled version of training methods or structural improvement in CNN. Experimental results show that target detection rates without homogeneous prior samples can reach 85%. Under these conditions, the traditional CNN model is invalid.

show abstract

Exploring the Potential of Conditional Adversarial Networks for Optical and SAR Image Matching

Abstract: This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any products or services.

Cited by 129 publications

References 29 publications

SAR-to-Optical Image Translation Based on Conditional Generative Adversarial Networks—Optimization, Opportunities and Limits

SAR-to-Optical Image Translation Based on Conditional Generative Adversarial Networks—Optimization, Opportunities and Limits

A Semi-Supervised Approach to Sar-Optical Image Matching

Transfer and Association: A Novel Detection Method for Targets without Prior Homogeneous Samples

Contact Info

Product

Resources

About