LoFTR: Detector-Free Local Feature Matching with Transformers

Sun, Jiaming; Shen, Zehong; Wang, Yuan; Bao, Hujun; Zhou, Xiaowei

doi:10.1109/cvpr46437.2021.00881

Cited by 700 publications

(477 citation statements)

References 28 publications

Supporting

Mentioning

372

Contrasting

Unclassified

Order By: Relevance

“…Jointly learning local feature description and matching. Several methods have recently proposed to jointly learn to compute and match descriptors [27,39,40,43,48]. All these methods use a siamese Convolutional Neural Network (CNN) to obtain dense local descriptors, but they significantly differ regarding the way they establish matches.…”

Section: Related Workmentioning

confidence: 99%

“…This 4D correlation tensor is then used as input to a second network that learns to modify it using soft-MNN and 4D convolutions. Instead of summarizing all the information into a 4D correlation tensor, the second category of methods [43,48] rely on so-called Transformers [10,15,19,26,37,52,55] to let the descriptors of both images communicate and adapt to each other. All these methods again focus on identifying correctly covisible correspondences and consider non-covisible correspondences as noise.…”

Section: Related Workmentioning

confidence: 99%

“…We identified the following limitations for our approach: 1 -The previous experiments showed that NeurHal is able to inpaint correspondences but the inpainted 8 R2D2 [38] + RE [13] SP+SG [43] + RE [13] LoFTR [48] + RE [13] DRCNet [27] + RE [13] NRE Feats. [24] + NRE [24] NeurHal + RE [13] NeurHal + NRE [ correspondence maps are much less peaked compared to the outpainted correspondence maps.…”

Section: Limitationsmentioning

confidence: 99%

“…LoFTR [48]+RE [13] DRCNet [27]+RE [13] NeurHal +NRE [24] Figure 10: Qualitative camera pose estimation results on low-overlap images from Scan-Net [17]: We show for every method keypoints used as input for the camera pose estimator in the source image (left image), along with their predicted reprojection in the target image (right image). We color-code keypoints based 2D spatial position in the source image.…”

Section: B2 Impact Of the Value Of γmentioning

confidence: 99%

See 3 more Smart Citations

Visual Correspondence Hallucination

Germain¹,

Lepetit²,

Bourmaud³

2021

Preprint

View full text Add to dashboard Cite

Given a pair of partially overlapping source and target images and a keypoint in the source image, the keypoint's correspondent in the target image can be either visible, occluded or outside the field of view. Local feature matching methods are only able to identify the correspondent's location when it is visible, while humans can also hallucinate its location when it is occluded or outside the field of view through geometric reasoning. In this paper, we bridge this gap by training a network to output a peaked probability distribution over the correspondent's location, regardless of this correspondent being visible, occluded, or outside the field of view. We experimentally demonstrate that this network is indeed able to hallucinate correspondences on unseen pairs of images. We also apply this network to a camera pose estimation problem and find it is significantly more robust than state-of-the-art local feature matching-based competitors. Code will be available at hugogermain.com/neurhal.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Limitationsmentioning

confidence: 99%

Section: B2 Impact Of the Value Of γmentioning

confidence: 99%

See 2 more Smart Citations

Visual Correspondence Hallucination

Germain¹,

Lepetit²,

Bourmaud³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…As various setups had been used for this evaluation [21,73,53,63], it is relatively difficult to make firm conclusions from Table 1. Even if almost in every setup RANSAC is employed for homography estimation, independent setups utilized different implementations (OpenCV and pydegensac).…”

Section: Homography Estimationmentioning

confidence: 99%

DFM: A Performance Baseline for Deep Feature Matching

Efe¹,

Ince²,

Alatan³

2021

Preprint

View full text Add to dashboard Cite

A novel image matching method is proposed that utilizes learned features extracted by an off-the-shelf deep neural network to obtain a promising performance. The proposed method uses pre-trained VGG architecture as a feature extractor and does not require any additional training specific to improve matching. Inspired by well-established concepts in the psychology area, such as the Mental Rotation paradigm, an initial warping is performed as a result of a preliminary geometric transformation estimate. These estimates are simply based on dense matching of nearest neighbors at the terminal layer of VGG network outputs of the images to be matched. After this initial alignment, the same approach is repeated again between reference and aligned images in a hierarchical manner to reach a good localization and matching performance. Our algorithm achieves 0.57 and 0.80 overall scores in terms of Mean Matching Accuracy (MMA) for 1 pixel and 2 pixels thresholds respectively on Hpatches dataset [4], which indicates a better performance than the state-of-the-art.

show abstract

A robust and automatic CT‐3D ultrasound registration method based on segmentation, context, and edge hybrid metric

Zhao

Dai

et al. 2023

Medical Physics

View full text Add to dashboard Cite

BackgroundThe fusion of computed tomography (CT) and ultrasound (US) image can enhance lesion detection ability and improve the success rate of liver interventional radiology. The image‐based fusion methods encounter the challenge of registration initialization due to the random scanning pose and limited field of view of US. Existing automatic methods those used vessel geometric information and intensity‐based metric are sensitive to parameters and have low success rate. The learning‐based methods require a large number of registered datasets for training.PurposeThe aim of this study is to provide a fully automatic and robust US‐3D CT registration method without registered training data and user‐specified parameters assisted by the revolutionary deep learning‐based segmentation, which can further be used for preparing training samples for the study of learning‐based methods.MethodsWe propose a fully automatic CT‐3D US registration method by two improved registration metrics. We propose to use 3D U‐Net‐based multi‐organ segmentation of US and CT to assist the conventional registration. The rigid transform is searched in the space of any paired vessel bifurcation planes where the best transform is decided by a segmentation overlap metric, which is more related to the segmentation precision than Dice coefficient. In nonrigid registration phase, we propose a hybrid context and edge based image similarity metric with a simple mask that can remove most noisy US voxels to guide the B‐spline transform registration. We evaluate our method on 42 paired CT‐3D US datasets scanned with two different US devices from two hospitals. We compared our methods with other exsiting methods with both quantitative measures of target registration error (TRE) and the Jacobian determinent with paired t‐test and qualitative registration imaging results.ResultsThe results show that our method achieves fully automatic rigid registration TRE of 4.895 mm, deformable registration TRE of 2.995 mm in average, which outperforms state‐of‐the‐art automatic linear methods and nonlinear registration metrics with paired t‐test's p value less than 0.05. The proposed overlap metric achieves better results than self similarity description (SSD), edge matching (EM), and block matching (BM) with p values of 1.624E‐10, 4.235E‐9, and 0.002, respectively. The proposed hybrid edge and context‐based metric outperforms context‐only, edge‐only, and intensity statistics‐only‐based metrics with p values of 0.023, 3.81E‐5, and 1.38E‐15, respectively. The 3D US segmentation has achieved mean Dice similarity coefficient (DSC) of 0.799, 0.724, 0.788, and precision of 0.871, 0.769, 0.862 for gallbladder, vessel, and branch vessel, respectively.ConclusionsThe deep learning‐based US segmentation can achieve satisfied result to assist robust conventional rigid registration. The Dice similarity coefficient‐based metrics, hybrid context, and edge image similarity metric contribute to robust and accurate registration.

show abstract

LoFTR: Detector-Free Local Feature Matching with Transformers

Cited by 700 publications

References 28 publications

Visual Correspondence Hallucination

Visual Correspondence Hallucination

DFM: A Performance Baseline for Deep Feature Matching

A robust and automatic CT‐3D ultrasound registration method based on segmentation, context, and edge hybrid metric

Contact Info

Product

Resources

About