FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence

Abstract

We present a descriptor, called fully convolutional selfsimilarity (FCSS), for dense semantic correspondence. To robustly match points among different instances within the same object class, we formulate FCSS using local selfsimilarity (LSS) within a fully convolutional network. In contrast to existing CNN-based descriptors, FCSS is inherently insensitive to intra-class appearance variations because of its LSS-based structure, while maintaining the precise localization ability of deep neural networks. The sampling patterns of local structure and the self-similarity measure are jointly learned within the proposed network in an end-to-end and multi-scale manner. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in existing image datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.

macJNet: weakly-supervised multimodal image deformable registration using joint learning framework and multi-sampling cascaded MIND

“…Yan et al [ 14 ] first proposed the adversarial image registration framework, which performs image registration tasks through a generator and evaluates the quality of the warped images by a discriminator. Kim et al [ 15 ] proposed a fully convolutional self-similarity to find dense semantic correspondence in mono-modality registration. A recent trend for multimodal image registration takes advantages of image to image translation [ 16 ], generative adversarial networks (GANs) convert the multimodal registration into a simpler unimodal task by learning transferable representations from multimodal images.…”

“…Kim et al proposed deep self-correlation (DSC) [ 24 ] to estimate cross-modal dense correspondences inspired by LSS and DSC has demonstrated its high accuracy on aligning multimodal image. Fully convolutional self-similarity (FCSS) [ 15 ] formulates LSS within a fully convolutional network to simultaneously learn the patch sampling patterns and self-similarity measures. Although FCSS dramatically improved performance for object-level semantic correspondence, it cannot deal with complex geometric variations, which frequently appears in medical image registration.…”