CATs++: Boosting Cost Aggregation with Convolutions and Transformers

Abstract

Cost aggregation is a highly important process in image matching tasks, which aims to disambiguate the noisy matching scores. Existing methods generally tackle this by hand-crafted or CNN-based methods, which either lack robustness to severe deformations or inherit the limitation of CNNs that fail to discriminate incorrect matches due to limited receptive fields and inadaptability. In this paper, we introduce Cost Aggregation with Transformers (CATs) to tackle this by exploring global consensus among initial correlation map with the help of some architectural designs that allow us to fully enjoy global receptive fields of self-attention mechanism. To this end, we include appearance affinity modeling, which helps to disambiguate the noisy initial correlation maps. Furthermore, we introduce some techniques, including multi-level aggregation to exploit rich semantics present at different feature levels and swapping self-attention to obtain reciprocal matching scores to act as strong regularization. Although competitive performance can be attained by CATs, it may face some limitations, i.e., high computational costs induced by the use of a standard transformer that its complexity grows with the size of spatial and feature dimensions, which restrict its applicability only at limited resolution and result in rather limited performance. To overcome this, we propose CATs++, an extension of CATs. Concretely, we introduce early convolutions prior to cost aggregation with a transformer to control the number of tokens as well as to inject some convolutional inductive bias, and propose a novel transformer architecture for both efficient and effective cost aggregation, which results in apparent performance boost and cost reduction. With the reduced costs, we manage to compose our network with a hierarchical structure to process higher-resolution inputs. With these combined, we conduct experiments to demonstrate the effectiveness of the proposed model over the latest methods. We evaluate the proposed method on standard benchmarks, including PF-WILLOW, PF-PASCAL, and SPair-71k. Our proposed methods outperform the previous state-of-the-art methods by large margins, setting a new state-of-the-art for all the benchmarks. We also provide extensive ablation studies and analyses.

Neural Matching Fields: Implicit Representation of Matching Fields for Visual Correspondence

“…7. We note the resolution which the method is evaluated, since [9,78] observe that the resolution of images affect the PCK performance, and the resolution of which the method outputs the correspondence field. It is shown that NeMF achieves competitive performance or even attains state-of-the-art performance for several alpha thresholds.…”

“…CHM [47] extends the PHM by employing high-dimensional convolutional kernels to aggregate 6D correlation maps. CATs [8] and its extension [9] use transformers [79,12] to explore global consensus from correlation maps thanks to transformers' ability to consider long-range interactions. All these works exploit rich semantics present at high-level features for robust matching across semantically similar images.…”

“…interpolation [66,65,26,8,78,9] or TPS warping with sparse keypoints [48,50,47], significantly reducing localization precision in matching details. Instead of these hand-crafted designs, several works [25,77,19] attempted to formulate a coarse-to-fine approach by utilizing multi-level features, but they often suffer from the propagation of initial error from the early coarse level.…”

Learning Complete and Discriminative Direction Pattern for Robust Palmprint Recognition

“…We believe that PCF-Net points out a novel direction for solving correspondence problems: learning reliable and geometric-invariant probabilistic coordinate representations. Future research directions include further optimization through cost aggregation [64] and graph matching [65].…”

College students’ perceptions of AI-based writing learning tools: With a focus on Google Translate, Naver Papago, and Grammarly

“…Due to these unconstrained settings, it should handle the additional challenges of large intra-class variations in appearance and background clutter. Recent deep learning-based matching models (Min et al 2019a;Liu et al 2020;Li et al 2020a;Li et al 2021;Zhao et al 2021;Min et al 2020;Cho et al 2021;Cho, Hong, and Kim 2022), following data-driven approach, were generally trained in a supervised fashion based on datasets (Ham et al ‡ Work done while at NAVER AI Lab. Correspondence to Dongyoon Han: dongyoon.han@navercorp.com.…”