Twin-Net Descriptor: Twin Negative Mining With Quad Loss for Patch-Based Matching

Abstract

Local keypoint matching is an important step for computer vision based tasks. In recent years, Deep Convolutional Neural Network (CNN) based strategies have been employed to learn descriptor generation to enhance keypoint matching accuracy. Recent state-of-art works in this direction primarily rely upon a triplet based loss function (and its variations) utilizing three samples: an anchor, a positive and a negative. In this work we propose a novel ''Twin Negative Mining'' based sampling strategy coupled with a Quad loss function to train a deep neural network based pipeline (Twin-Net) for generating a robust descriptor that provides an increased discriminatory power to differentiate between patches that do not correspond to each other. Our sampling strategy and choice of loss function is aimed at placing an upper bound that descriptors of two patches representing same location could be at worst no more dissimilar than the descriptors of two similar looking patches that do-not belong to same 3D location. This results in an increase in the generalization capability of the network and outperforms its existing counterparts when trained over the same datasets. Twin-Net outputs a 128-dimensional descriptor and uses L 2 Distance as the similarity metric, and hence conforms to the classical descriptor matching pipelines such as that of SIFT. Our results on Brown and HPatches datasets demonstrate Twin-Net's consistently better performance as well as better discriminatory and generalization capability as compared to the state-of-art.INDEX TERMS Descriptor learning, twin negative sampling, patch matching, quad loss.

Paying Attention to Multiscale Feature Maps in Multimodal Image Matching

“…Our approach achieved a new SOTA performance when trained on the Liberty and Yosemite datasets, surpassing the previous SOTA by 24%. When trained on Notredame our approach is slightly outperformed by CS L2-Net [36] and Twin-Net [20]. We suspect it is due to the significant geometric deformations in Notredame, degrading the feature maps extracted by our Siamese CNN.…”

“…The scheme combines hard negative and positive mining inspired by Mishchuk et al [26] and Simo et al [34], respectively. Irshad et al followed this line with the Twin-Net [20] approach, by proposing to mine twin negatives along with a dedicated Quad Loss, designed for single modality patches. In twin negatives mining, the first negative is mined as in Mishchuk et al [26], and its closest negative is picked as the second negative.…”

“…The results are presented in Table 4. We quote the previous results reported on this benchmark by [38,20]. 3 Benchmark [14] consisting of three multimodal datasets sampled on a uniform grid layout.…”

“…We suspect it is due to the significant geometric deformations in Notredame, degrading the feature maps extracted by our Siamese CNN. Notredame also con- sists of some extremely hard samples, that Twin-Net [20] might be able to mine better using its improved hard samples mining scheme.…”

“…Contrastive [34,6] and Triplet Losses [26] were used along with Euclidean distance to learn descriptors encoding latent space similarity. Due to the high ratio of negative pairs, most of which being easy negatives barely contributing to the loss, hard negative mining schemes have become prevalent in recent studies [5,26,6,20], sometimes combined with hard positive mining [34,4,38].…”