PARN: Pyramidal Affine Regression Networks for Dense Semantic Correspondence

Abstract

This paper presents a deep architecture for dense semantic correspondence, called pyramidal affine regression networks (PARN), that estimates locally-varying affine transformation fields across images. To deal with intra-class appearance and shape variations that commonly exist among different instances within the same object category, we leverage a pyramidal model where affine transformation fields are progressively estimated in a coarse-to-fine manner so that the smoothness constraint is naturally imposed within deep networks. PARN estimates residual affine transformations at each level and composes them to estimate final affine transformations. Furthermore, to overcome the limitations of insufficient training data for semantic correspondence, we propose a novel weakly-supervised training scheme that generates progressive supervisions by leveraging a correspondence consistency across image pairs. Our method is fully learnable in an end-to-end manner and does not require quantizing infinite continuous affine transformation fields. To the best of our knowledge, it is the first work that attempts to estimate dense affine transformation fields in a coarse-to-fine manner within deep networks. Experimental results demonstrate that PARN outperforms the state-of-the-art methods for dense semantic correspondence on various benchmarks.

Show, Match and Segment: Joint Weakly Supervised Learning of Semantic Matching and Object Co-Segmentation

“…Motivated by the procedure of non-rigid image registration between an image pair, the RTNs [40] use recurrent networks to progressively compute dense correspondences between two images. In addition to estimating geometric transformations between an image pair [18], [38], [40], another line of research focuses on establishing dense per-pixel correspondences without using any geometric models [39]. Rocco Addressing semantic matching (left) or object co-segmentation (right) in isolation often suffers from the effect of background clutters (for semantic matching) or only focuses on segmenting the discriminative parts (for object co-segmentation).…”

“…Although our method performs slightly worse than the OHG [30] on the PASCAL category, the OHG method uses additional images from the PASCAL VOC 2007 dataset. In the bottom block of Table 1, we follow the PARN [38] and RTNs [40], and resize all images to the larger dimension to 100 (i.e., resizing max(H, W ) to 100). The proposed method also performs favorably against all competing methods.…”

Semantic Attribute Matching Networks

“…To deal with locally-varying geometric deformations, some methods such as UCN [7] and CAT-FCSS [25] were proposed based on STNs [18]. Recently, PARN [19], NC-Net [43], and RTNs [23] were proposed to estimate locally-varying transformation fields using a coarse-to-fine scheme [19], neighbourhood consensus [43], and an iteration technique [23]. These methods [19,43,23] presume that the attribute variations between source and target images are negligible in the deep feature space.…”

“…Recently, PARN [19], NC-Net [43], and RTNs [23] were proposed to estimate locally-varying transformation fields using a coarse-to-fine scheme [19], neighbourhood consensus [43], and an iteration technique [23]. These methods [19,43,23] presume that the attribute variations between source and target images are negligible in the deep feature space. However, in practice the deep features often show limited performance in handling different attributes.…”

“…Semantic Matching Figure 2. Intuition of SAM-Net: (a) methods for semantic matching [41,42,23,19], (b) methods for attribute transfer [11,21,28], and (c) SAM-Net, which recurrently weaves the advantages of both existing semantic matching and attribute transfer techniques.…”