RFNet: Unsupervised Network for Mutually Reinforcing Multi-modal Image Registration and Fusion

Xu, Han; Ma, Jiayi; Yuan, Jiteng; Le, Zhuliang; Liu, Wei

doi:10.1109/cvpr52688.2022.01906

Cited by 55 publications

(21 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Two-view corresponding is a fundamental problem in computer vision. It aims to establish sparse feature correspondences/matches between two-view images and estimate geometry relationship, serving as a premise for many complex vision problems such as structure from motion (Snavely, Seitz, and Szeliski 2008), simultaneous location and mapping (Mur-Artal, Montiel, and Tardos 2015), visual localization (Philbin et al 2010), and image fusion (Xu et al 2022). The most classical geometry matching pipeline starts from feature extraction and matching, and great efforts have been spent on handcrafted or learning-based detectors and descriptors, e.g., SIFT (Lowe 2004) and SuperPoint (DeTone, Malisiewicz, and Rabinovich 2018).…”

Section: Introductionmentioning

confidence: 99%

ConvMatch: Rethinking Network Design for Two-View Correspondence Learning

Zhang

2023

AAAI

View full text Add to dashboard Cite

Multilayer perceptron (MLP) has been widely used in two-view correspondence learning for only unordered correspondences provided, and it extracts deep features from individual correspondence effectively. However, the problem of lacking context information limits its performance and hence, many extra complex blocks are designed to capture such information in the follow-up studies. In this paper, from a novel perspective, we design a correspondence learning network called ConvMatch that for the first time can leverage convolutional neural network (CNN) as the backbone to capture better context, thus avoiding the complex design of extra blocks. Specifically, with the observation that sparse motion vectors and dense motion field can be converted into each other with interpolating and sampling, we regularize the putative motion vectors by estimating dense motion field implicitly, then rectify the errors caused by outliers in local areas with CNN, and finally obtain correct motion vectors from the rectified motion field. Extensive experiments reveal that ConvMatch with a simple CNN backbone consistently outperforms state-of-the-arts including MLP-based methods for relative pose estimation and homography estimation, and shows promising generalization ability to different datasets and descriptors. Our code is publicly available at https://github.com/SuhZhang/ConvMatch.

show abstract

Section: Introductionmentioning

confidence: 99%

ConvMatch: Rethinking Network Design for Two-View Correspondence Learning

Zhang

2023

AAAI

View full text Add to dashboard Cite

show abstract

“…10,29 Deep learning-based methods, such as RFNet, AT-GAN, and SemLA, utilize deep neural networks to promote multimodal image registration accuracy. 15,36,37 However, deep learning-based methods depend greatly on the quality of training data, which limits their performance in the registration of coral reefs with less texture.…”

Section: Introductionmentioning

confidence: 99%

“… – 12 Generally, there are four types of automatic registration methods: intensity-based methods, feature-based methods, coarse-to-fine methods, and deep learning-based methods 13 – 15 Intensity-based methods, such as grayscale matching, match images by comparing reflection differences between pixels within correlation windows in images to be matched. These methods require remote sensing images with high consistency in terms of scale and orientation 9 .…”

Section: Introductionmentioning

confidence: 99%

“…For example, SIFT was utilized to guarantee preregistration results close to the ground truth, and mutual information was utilized during the fine-tuning process to achieve the most precise registration results 10 , 29 . Deep learning-based methods, such as RFNet, AT-GAN, and SemLA, utilize deep neural networks to promote multimodal image registration accuracy 15 , 36 , 37 . However, deep learning-based methods depend greatly on the quality of training data, which limits their performance in the registration of coral reefs with less texture.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automatic registration method for medium-resolution remote sensing images of coral reefs with morphological information pairing and constrained iterative fining

Chen,

Pian,

Chen

et al. 2023

J. Appl. Rem. Sens.

View full text Add to dashboard Cite

“…Moreover, there are many methods based on deep learning, such as Mu-net [18], PCNet [19], RFNet [20] and Fourier-Net [21]. In order to adapt various types of multimodal images, Mu-Net uses the structural similarity to design a loss function that allows Mu-net to achieve comprehensive and accurate registration [18].…”

Section: Introductionmentioning

confidence: 99%

LPHOG: A Line Feature and Point Feature Combined Rotation Invariant Method for Heterologous Image Registration

He,

Jiang,

Hao

et al. 2023

Remote Sensing

View full text Add to dashboard Cite

Remote sensing image registration has been a very important research topic, especially the registration of heterologous images. In the research of the past few years, numerous registration algorithms for heterogenic images have been developed, especially feature-based matching algorithms, such as point feature-based or line feature-based matching methods. However, there are few matching algorithms that combine line and point features. Therefore, this study proposes a matching algorithm that combines line features and point features while achieving good rotation invariance. It comprises LSD detection of line features, keypoint extraction, and HOG-like feature descriptor construction. The matching performance is compared with state-of-the-art matching algorithms on three heterogeneous image datasets (optical–SAR dataset, optical–infrared dataset, and optical–optical dataset), verifying our method’s rotational invariance by rotating images in each dataset. Finally, the experimental results show that our algorithm outperforms the state-of-the-art algorithms in terms of matching performance while possessing very good rotation invariance.

show abstract

RFNet: Unsupervised Network for Mutually Reinforcing Multi-modal Image Registration and Fusion

Cited by 55 publications

References 30 publications

ConvMatch: Rethinking Network Design for Two-View Correspondence Learning

ConvMatch: Rethinking Network Design for Two-View Correspondence Learning

Automatic registration method for medium-resolution remote sensing images of coral reefs with morphological information pairing and constrained iterative fining

LPHOG: A Line Feature and Point Feature Combined Rotation Invariant Method for Heterologous Image Registration

Contact Info

Product

Resources

About