Exploiting Deep Matching and SAR Data for the Geo-Localization Accuracy Improvement of Optical Satellite Images

Merkle, Nina; Luo, Wenjie; Auer, Stefan; Müller, Rupert; Urtasun, Raquel

doi:10.3390/rs9060586

Cited by 97 publications

(78 citation statements)

References 49 publications

(72 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This sensor provides extremely high geo-location accuracy [46] and thus perspectives to retrieve highly accurate 3D point information via radargrammetric processing [47][48][49]. Then such GCPs can be transferred into the optical Pléiades image via multi-modal image matching as presented, for instance, in References [50][51][52].…”

Section: Sensor Modeling and Parameter Optimizationmentioning

confidence: 99%

Mapping with Pléiades—End-to-End Workflow

2019

View full text Add to dashboard Cite

In this work, we introduce an end-to-end workflow for very high-resolution satellite-based mapping, building the basis for important 3D mapping products: (1) digital surface model, (2) digital terrain model, (3) normalized digital surface model and (4) ortho-rectified image mosaic. In particular, we describe all underlying principles for satellite-based 3D mapping and propose methods that extract these products from multi-view stereo satellite imagery. Our workflow is demonstrated for the Pléiades satellite constellation, however, the applied building blocks are more general and thus also applicable for different setups. Besides introducing the overall end-to-end workflow, we need also to tackle single building blocks: optimization of sensor models represented by rational polynomials, epipolar rectification, image matching, spatial point intersection, data fusion, digital terrain model derivation, ortho rectification and ortho mosaicing. For each of these steps, extensions to the state-of-the-art are proposed and discussed in detail. In addition, a novel approach for terrain model generation is introduced. The second aim of the study is a detailed assessment of the resulting output products. Thus, a variety of data sets showing different acquisition scenarios are gathered, allover comprising 24 Pléiades images. First, the accuracies of the 2D and 3D geo-location are analyzed. Second, surface and terrain models are evaluated, including a critical look on the underlying error metrics and discussing the differences of single stereo, tri-stereo and multi-view data sets. Overall, 3D accuracies in the range of 0.2 to 0.3 m in planimetry and 0.2 to 0.4 m in height are achieved w.r.t. ground control points. Retrieved surface models show normalized median absolute deviations around 0.9 m in comparison to reference LiDAR data. Multi-view stereo outperforms single stereo in terms of accuracy and completeness of the resulting surface models.then results in 3D coordinates (i.e., a 3D point cloud) that is resampled into an intermediate DSM. Second, those DSMs are fused and post-processed yielding the final DSM. Within this task, the characteristics of the underlying sensor system is exploited, like availability of image triplets in case of Pléiadesand including high-level processing techniques for the given input. The applicability of high-level processing options, like the semi-global matching (SGM) technique, is investigated, enhanced and resulting achievements are illustrated and validated. 3.In the third task, one DTM and one nDSM are generated solely based on the DSM produced in the previous task. The main steps are ground point filtering, detecting points in the given DSM which are located on bare earth, hole filling to interpolate non-ground information and post-processing mainly for outlier removal. In this way, we can (a) filter man-made structures and vegetation and (b) retrieve vegetation or building heights, finally yielding the DTM and nDSM models. While this is a well-established procedure for airborne light dete...

show abstract

Section: Sensor Modeling and Parameter Optimizationmentioning

confidence: 99%

Mapping with Pléiades—End-to-End Workflow

2019

View full text Add to dashboard Cite

show abstract

Section: Deep Learning For Sar-optical Matchingmentioning

confidence: 99%

“…The first notable examples of this were provided in short succession by (Merkle et al, 2017b) and (Mou et al, 2017) who both proposed variants of a 2-stream architecture. (Merkle et al, 2017b) trained a siamese network to predict the relative shift between SAR and optical patches in order to improve the geolocalization accuracy of the optical data, while (Mou et al, 2017) trained a pseudo-siamese variant as a binary correspondence classifier. Taking inspiration from these seminal works, we extended the network proposed by (Mou et al, 2017) by enhancing the feature fusion stage and converting the output to a similarity score based on the soft-max probability (Hughes et al, 2018b).…”

Section: Deep Learning For Sar-optical Matchingmentioning

confidence: 99%

A Semi-Supervised Approach to Sar-Optical Image Matching

Hughes

Schmitt

2019

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

<p><strong>Abstract.</strong> Matching synthetic aperture radar (SAR) and optical remote sensing imagery is a key first step towards exploiting the complementary nature of these data in data fusion frameworks. While numerous signal-based approaches to matching have been proposed, they often fail to perform well in multi-sensor situations. In recent years deep learning has become the go-to approach for solving image matching in computer vision applications, and has also been adapted to the case of SAR-optical image matching. However, the hitherto proposed techniques still fail to match SAR and optical imagery in a generalizable manner. These limitations are largely due to the complexities in creating large-scale datasets of corresponding SAR and optical image patches. In this paper we frame the matching problem within semi-supervised learning, and use this as a proxy for investigating the effects of data scarcity on matching. In doing so we make an initial contribution towards the use of semi-supervised learning for matching SAR and optical imagery. We further gain insight into the non-complementary nature of commonly used supervised and unsupervised loss functions, as well as dataset size requirements for semi-supervised matching.</p>

show abstract

“…Note that tie points are the common points between the SAR and optical images, and can be obtained by manual or automatic sparse matching between two images. Since the automation of the tie point generation process is not the focus of this study, we refer the reader to possible solutions described in [36,37,38]. The geographic coordinates of the tie points in the SAR image are calculated by the inverse rational functions computed for the SAR imagery as described in Section 2.1:…”

Section: Sar-optical Multi-sensor Block Adjustmentmentioning

confidence: 99%

A framework for SAR-optical stereogrammetry over urban areas

Bagheri

Schmitt

d’Angelo

et al. 2018

ISPRS Journal of Photogrammetry and Remote Sensing

View full text Add to dashboard Cite

Currently, numerous remote sensing satellites provide a huge volume of diverse earth observation data. As these data show different features regarding resolution, accuracy, coverage, and spectral imaging ability, fusion techniques are required to integrate the different properties of each sensor and produce useful information. For example, synthetic aperture radar (SAR) data can be fused with optical imagery to produce 3D information using stereogrammetric methods. The main focus of this study is to investigate the possibility of applying a stereogrammetry pipeline to very-high-resolution (VHR) SAR-optical image pairs. For this purpose, the applicability of semi-global matching is investigated in this unconventional multi-sensor setting. To support the image matching by reducing the search space and accelerating the identification of correct, reliable matches, the possibility of establishing an epipolarity constraint for VHR SAR-optical image pairs is investigated as well. In addition, it is shown that the absolute geolocation accuracy of VHR optical imagery with respect to VHR SAR imagery such as provided by TerraSAR-X can be improved by a multi-sensor block adjustment formulation based on rational polynomial coefficients. Finally, the feasibility of generating point clouds with a median accuracy of about 2 m is demonstrated and confirms the potential of 3D reconstruction from SAR-optical image pairs over urban areas.

show abstract

Exploiting Deep Matching and SAR Data for the Geo-Localization Accuracy Improvement of Optical Satellite Images

Cited by 97 publications

References 49 publications

Mapping with Pléiades—End-to-End Workflow

Mapping with Pléiades—End-to-End Workflow

A Semi-Supervised Approach to Sar-Optical Image Matching

A framework for SAR-optical stereogrammetry over urban areas

Contact Info

Product

Resources

About