Cascade Network with Guided Loss and Hybrid Attention for Finding Good Correspondences

Abstract

Finding good correspondences is a critical prerequisite in many feature based tasks. Given a putative correspondence set of an image pair, we propose a neural network which finds correct correspondences by a binary-class classifier and estimates relative pose through classified correspondences. First, we analyze that due to the imbalance in the number of correct and wrong correspondences, the loss function has a great impact on the classification results. Thus, we propose a new Guided Loss that can directly use evaluation criterion (Fn-measure) as guidance to dynamically adjust the objective function during training. We theoretically prove that the perfect negative correlation between the Guided Loss and Fn-measure, so that the network is always trained towards the direction of increasing Fn-measure to maximize it. We then propose a hybrid attention block to extract feature, which integrates the Bayesian attentive context normalization (BACN) and channel-wise attention (CA). BACN can mine the prior information to better exploit global context and CA can capture complex channel context to enhance the channel awareness of the network. Finally, based on our Guided Loss and hybrid attention block, a cascade network is designed to gradually optimize the result for more superior performance. Experiments have shown that our network achieves the state-of-the-art performance on benchmark datasets. Our code will be available in https://github.com/wenbingtao/GLHA.

A Survey on the Deep Learning-Based Mismatch Removal: Principles and Methods

“…The former operates with features and fine-tuned features as input and output, while for the latter, the input and output are correspondences and pruned correspondences, respectively (schematic illustrations of the two modes are shown in Figure 4). For instance, in order to address the imbalance between inliers and outliers, Guided Loss and Hybrid Attention (GLHA) [84] constructs a coarseto-fine cascade network by using attention mechanism; Consensus Learning Network (CLNet) [83] learns a network by progressively pruning the correspondences; Interactive Generative Structure Network (IGS-Net) [85] captures the coarse-to-fine transformations of matches through progressive representation learning. In summary, for the purpose of designing a deeper network, most of the IMBs are organized in the one-shot mode.…”

“…The "−" stream is organized as the linear structure, and the "|" stream receives the outputs of every IMB in the "−" stream and finally outputs the features of matches (the schematic illustration of the "T" structure is shown in Figure 5). "T" structure network was firstly introduced by T-Net [84]. It adopts "−" stream to iteratively learn features of matches and another "|" stream to integrate the features and generate matching probabilities.…”

“…While in shallow networks, the weights cannot be effectively learned due to the less distinguishable output features of the networks. To address this limitation, the improved version of ACN, Bayesian Attentive Context Normalization (BACN) [84], was introduced. Whereas, BACN employs NNDR [71] to compensate for the dilemma of weight learning in shallow networks, which are not available if matches are not generated from feature matching.…”

“…Classification loss Lcls usually is the binary cross entropy of labels and predications. Directly using the binary cross entropy as the training loss will lead to the network being biased negative class since the number of outliers is generally much more than that of outliers [84]. Considering the bias problem, BACN [84] proposes to minimize Instance Balance Cross Entropy Loss (IB-CE-Loss) 1 1…”