SOSNet: Second Order Similarity Regularization for Local Descriptor Learning

Abstract

Despite the fact that Second Order Similarity (SOS) has been used with significant success in tasks such as graph matching and clustering, it has not been exploited for learning local descriptors. In this work, we explore the potential of SOS in the field of descriptor learning by building upon the intuition that a positive pair of matching points should exhibit similar distances with respect to other points in the embedding space. Thus, we propose a novel regularization term, named Second Order Similarity Regularization (SOSR), that follows this principle. By incorporating SOSR into training, our learned descriptor achieves state-of-the-art performance on several challenging benchmarks containing distinct tasks ranging from local patch retrieval to structure from motion. Furthermore, by designing a von Mises-Fischer distribution based evaluation method, we link the utilization of the descriptor space to the matching performance, thus demonstrating the effectiveness of our proposed SOSR. Extensive experimental results, empirical evidence, and in-depth analysis are provided, indicating that SOSR can significantly boost the matching performance of the learned descriptor.

Kornia: an Open Source Differentiable Computer Vision Library for PyTorch

“…Kornia provides operators to detect local features, compute descriptors, and perform feature matching. The module contains differentiable versions of the Harris corner detector [30], Shi-Tomasi corner detector detector [31], Hessian detector [32], their scale and affine covariant versions [33], DoG [13], patch dominant gradient orientation [13] and the SIFT descriptor [13] and recent deep learning based methods such as HardNet [34] or SOSNet [35]. In addition, this module provides a high level API to perform detections in scale-space, where classical hard non-maxima suppression is replaced with its soft version, similar to the recently proposed Multiscale Index Proposal layer (M-SIP) [36], one can seamlessly replace any or all modules with deep learned counterparts.…”

On Aggregation of Unsupervised Deep Binary Descriptor With Weak Bits

“…Subsequently, L2-NET [60] trains a Siamese network for pairwised patches and produces binary codes by directly quantizing the real-valued outputs, where different regularization terms are applied on the intermediate layer outputs to improve the code quality. In [61], Second Order Similarity Regularization (SOSR) is incorporated into the proposed SOSNet as a regularization term to boost the matching performance. In [30], they train a deep network termed DN4, where a local descriptor is learned based on image-to-class measure.…”

D-MSCD: Mean-Standard Deviation Curve Descriptor Based on Deep Learning

“…Zhang et al [17] proposed a new triplet loss based on HardNet, which replaced the hard margin with dynamic soft margin, and got a better matching performance. Tian et al [18] proposed the Second Order Similarity Regularization (SOSR) and incorporated second order similarities into the learning of local descriptors. The matching performance of learning descriptors is significantly improved.…”

“…With the continuous success of deep learning in image recognition tasks, the current research on image feature matching has entered a new data-driven era. In recent years, attempts on using deep learning for image feature description and matching have also shown great opportunities [11]- [18]. However, there are no reports on the method using deep learning to describe curve features at present.…”