Optical and SAR Image Dense Registration Using a Robust Deep Optical Flow Framework

Abstract

The co-registration of optical and SAR imageries is the bottleneck in exploring the complementary information from the two multi-modal datasets. The difficulties lie in not only the complex radiometric relationship between them, but also the distinct geometrical models of the optical and SAR imaging systems, which cause it nontrivial to explicitly depict the spatial relationship between the corresponding image regions when elevation fluctuations exist. This article aims to investigate the optical flow technique for the pixelwise dense registration of the high resolution optical and SAR images, so as to get rid of the outlier removal and geometric mapping procedures, which have to be conducted in the classical image registration approaches that are based on sparse feature point matching. Herein, a deep optical flow framework is designed. Firstly, a dilated feature concatenation method is proposed to enhance the discriminability of the pixelwise features for similarity measurement. An effective network training strategy is used, based on a smoothed flow loss, and also a training dataset that contains simulated elevation fluctuations. Secondly, we propose a self-supervised optical flow finetuning strategy. It incorporates the strength of the blockwise matching approach, which produces better matching precision, into the proposed pixelwise matching procedure. In this way, the accuracy of the optical-SAR dense registration is substantially improved. Extensive experiments conducted on the 1m resolution optical-SAR image pairs of different land-cover types and distinct topographic conditions indicate that, the proposed OSFlowNet-Ft framework is quite robust, and has the potential to perform the optical-SAR image dense registration in practical applications. The Python code of the proposed deep optical flow network will be made available.

Registration of Large Optical and SAR Images with Non-Flat Terrain by Investigating Reliable Sparse Correspondences

“…Then, two local small patches surrounding p and q s are cut from the SAR and optical images, respectively, termed as J S and J O . The dense matching is conducted on {J S , J O } using the OSFlowNet, which is a learning-based optical-SAR flow framework proposed in our previous research [34], resulting in the pixelwise displacement map F. The 2D displacement vector of the central pixel p can be obtained from F, termed as f p x , f p y . If the template-based sparse matching is successful, the dense matching result would also be valid with a high probability, considering that the probable large initial displacement has been removed by the sparse matching process.…”

“…Here is a brief introduction to the deep learning-based OSFlowNet proposed in our previous work [34]. It uses a two-branched pseudo-Siamese network for optical and SAR pixelwise feature extraction and then produces a 4D correlation volume for feature similarity measurement.…”

“…Also, the DEM (digital elevation modal) images used for the geometric calibration are usually of low spatial and elevation resolution. Several pixelwise dense registration approaches based on the optical flow technique have been proposed to deal with the local geometric distortion problem [32][33][34]. However, the pixelwise registration would have a high memory demand.…”

Robust Registration of Optical and SAR Images Using Multi-Orientation Relative Total Variation Structural Representation

“…Unlike these traditional methods, the learning-based methods employ deep convolutional networks to extract more feature representations from images, which have made dramatic progress on multimodal image registration [36], [37]. Ma et al [38] proposed a robust two-step registration method for multimodal images, in which the convolutional neural network (CNN) features were combined with local features for feature matching.…”

Multifeature Semantic Complementation Network for Marine Oil Spill Localization and Segmentation Based on SAR Images

“…Satellite remote sensing is widely used in marine oil spill detection due to its comprehensive coverage and high timeliness. In recent years, related researchers have conducted studies in satellite remote sensing [6], [7]. Synthetic aperture radar (SAR) has become a powerful tool for oil spill detection because of its all-day monitoring and unaffected capability by clouds and fog [8]- [10].…”