Explore Better Network Framework for High-Resolution Optical and SAR Image Matching

Zhang, Han; Lei, Lin; Ni, Weijian; Tang, Tao; Jian, Wu; Xiang, Deliang; Kuang, Gangyao

doi:10.1109/tgrs.2021.3126939

Cited by 18 publications

(37 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To qualitatively evaluate the structural features of the pseudo-optical images generated by our network, inspired by the research in [61], a fast Fourier transform (FFT)-accelerated sum of squared differences (SSD) method is used to measure the similarity between the structural features of pseudo-optical images obtained via different methods and those of real optical images. The value of the SSD score plot indicates the offset between image pairs, and a smaller value indicates a higher similarity of their features [62]. In addition, it is noted that the SSD score map obtained with the maximum offset set to 8 pixels has dimensions of 17 × 17.…”

Section: Discussionmentioning

confidence: 99%

A Dual-Generator Translation Network Fusing Texture and Structure Features for SAR and Optical Image Matching

Nie

Tang

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

The matching problem for heterologous remote sensing images can be simplified to the matching problem for pseudo homologous remote sensing images via image translation to improve the matching performance. Among such applications, the translation of synthetic aperture radar (SAR) and optical images is the current focus of research. However, the existing methods for SAR-to-optical translation have two main drawbacks. First, single generators usually sacrifice either structure or texture features to balance the model performance and complexity, which often results in textural or structural distortion; second, due to large nonlinear radiation distortions (NRDs) in SAR images, there are still visual differences between the pseudo-optical images generated by current generative adversarial networks (GANs) and real optical images. Therefore, we propose a dual-generator translation network for fusing structure and texture features. On the one hand, the proposed network has dual generators, a texture generator, and a structure generator, with good cross-coupling to obtain high-accuracy structure and texture features; on the other hand, frequency-domain and spatial-domain loss functions are introduced to reduce the differences between pseudo-optical images and real optical images. Extensive quantitative and qualitative experiments show that our method achieves state-of-the-art performance on publicly available optical and SAR datasets. Our method improves the peak signal-to-noise ratio (PSNR) by 21.0%, the chromatic feature similarity (FSIMc) by 6.9%, and the structural similarity (SSIM) by 161.7% in terms of the average metric values on all test images compared with the next best results. In addition, we present a before-and-after translation comparison experiment to show that our method improves the average keypoint repeatability by approximately 111.7% and the matching accuracy by approximately 5.25%.

show abstract

Section: Discussionmentioning

confidence: 99%

A Dual-Generator Translation Network Fusing Texture and Structure Features for SAR and Optical Image Matching

Nie

Tang

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…The area-based methods register the image pairs using a similarity metric function, while the feature-based methods include four processes: feature extraction, feature matching, transformation model estimation and image re-sampling and warping. As deep learning has great potential in feature extraction, numerous researchers have designed data-driven strategy in the field of cross-modal image alignment [35,20,4]. Although image alignment is a necessary step in many fields, it brings extra time consumption and cannot completely address the weakly misalignment problem.…”

Section: Cross-modal Image Alignmentmentioning

confidence: 99%

Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection

Yuan¹,

Wang²,

Wei³

2022

Preprint

View full text Add to dashboard Cite

Integrating multispectral data in object detection, especially visible and infrared images, has received great attention in recent years. Since visible (RGB) and infrared (IR) images can provide complementary information to handle light variations, the paired images are used in many fields, such as multispectral pedestrian detection, RGB-IR crowd counting and RGB-IR salient object detection. Compared with natural RGB-IR images, we find detection in aerial RGB-IR images suffers from cross-modal weakly misalignment problems, which are manifested in the position, size and angle deviations of the same object. In this paper, we mainly address the challenge of cross-modal weakly misalignment in aerial RGB-IR images. Specifically, we firstly explain and analyze the cause of the weakly misalignment problem. Then, we propose a Translation-Scale-Rotation Alignment (TSRA) module to address the problem by calibrating the feature maps from these two modalities. The module predicts the deviation between two modality objects through an alignment process and utilizes Modality-Selection (MS) strategy to improve the performance of alignment. Finally, a two-stream feature alignment detector (TSFADet) based on the TSRA module is constructed for RGB-IR object detection in aerial images. With comprehensive experiments on the public DroneVehicle datasets, we verify that our method reduces the effect of the cross-modal misalignment and achieve robust detection results.

show abstract

“…Figures [6][7][8][9][10][11] show the corresponding points matched by the proposed method and other four matching networks on OS data. A yellow line represents a pair of correct matched points, while a red line represents false matching.…”

Section: Qualitative Experimentsmentioning

confidence: 99%

“…Due to different imaging principles, there are nonlinear radiometric and geometric differences between optical and SAR images. Furthermore, speckle noise on SAR images seriously affects matching performance, which makes it very challenging to find the conjugate features for matching of optical and SAR images [6]. For optical and SAR image matching, the difficulty lies in constructing robust features from heterogeneous image pairs.…”

Section: Introductionmentioning

confidence: 99%

“…To deal with this problem, many studies have been conducted using the traditional handcrafted features for matching [7][8][9]. However, considering that handcrafted feature descriptors are usually constructed by image features under a small local image region, such as the intensity or gradient value, it is not effective to deal with the large disparity between the local geometric structures in optical and SAR images [6]. Therefore, some recent studies turned to deep learning with the development of artificial intelligence techniques.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PM-Net: A Multi-Level Keypoints Detector and Patch Feature Learning Network for Optical and SAR Image Matching

Chen

2022

Applied Sciences

View full text Add to dashboard Cite

Due to the differences in radiation and geometric characteristics of optical and synthetic aperture radar (SAR) images, there is still a huge challenge for accurate matching. In this paper, we propose a patch-matching network (PM-Net) to improve the matching performance of optical and SAR images. First, a multi-level keypoints detector (MKD) with fused high-level and low-level features is presented to extract more robust keypoints from optical and SAR images. Second, we use a two-channel network structure to improve the image patch matching performance. Benefiting from this design, the proposed method can directly learn the similarity between optical and SAR image patches without manually designing features and descriptors. Finally, the MKD and two-channel net-work are trained separately on GL3D and QXS-SAROPT data sets, and the PM-Net is tested on multiple pairs of optical and SAR images. The experimental results demonstrate that the proposed method outperforms four advanced image matching networks on qualitative and quantitative assessments. The quantitative experiment results show that using our method correct matching points numbers are increased by more than 1.15 times, the value of F1-measure is raised by an average of 7.4% and the root mean squared error (RMSE) is reduced by more than 15.3%. The advantages of MKD and the two-channel network are also verified through ablation experiments.

show abstract

Explore Better Network Framework for High-Resolution Optical and SAR Image Matching

Cited by 18 publications

References 50 publications

A Dual-Generator Translation Network Fusing Texture and Structure Features for SAR and Optical Image Matching

A Dual-Generator Translation Network Fusing Texture and Structure Features for SAR and Optical Image Matching

Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection

PM-Net: A Multi-Level Keypoints Detector and Patch Feature Learning Network for Optical and SAR Image Matching

Contact Info

Product

Resources

About