Dynamic Context Correspondence Network for Semantic Alignment

Abstract

Establishing semantic correspondence is a core problem in computer vision and remains challenging due to large intra-class variations and lack of annotated data. In this paper, we aim to incorporate global semantic context in a flexible manner to overcome the limitations of prior work that relies on local semantic representations. To this end, we first propose a context-aware semantic representation that incorporates spatial layout for robust matching against local ambiguities. We then develop a novel dynamic fusion strategy based on attention mechanism to weave the advantages of both local and context features by integrating semantic cues from multiple scales. We instantiate our strategy by designing an end-to-end learnable deep network, named as Dynamic Context Correspondence Network (DCCNet). To train the network, we adopt a multi-auxiliary task loss to improve the efficiency of our weakly-supervised learning procedure. Our approach achieves superior or competitive performance over previous methods on several challenging datasets, including PF-Pascal, PF-Willow, and TSS, demonstrating its effectiveness and generality. Feature Extractor

PGA-SiamNet: Pyramid Feature-Based Attention-Guided Siamese Network for Remote Sensing Orthoimagery Building Change Detection

“…Recently, weakly supervised approaches have been proposed in an attempt to compensate for the dependence on the large annotated data, such as training with synthetic data obtained by a given geometric transformation [32][33][34]. However, detecting building changes with these methods has some restrictions: either an accurate position of the change cannot be given, or information of an independent building is required.…”

Semi-Global Context Network for Semantic Correspondence

“…The global semantic context or set of contextual information on inter-pixel relations within an image, can be used to help a visual system recognize an object's spatial layout to indicate where and how the object appears in the image. Recently [22], the global semantic context has been incorporated into the NC-Net, further improving its performance by enabling robust matching against repetitive patterns and intra-class variation. In this approach, contextaware features that contain the global semantic context are generated and then correlation maps derived from local and context-aware features are fused to apply the global semantic context to local features.…”

“…In this approach, contextaware features that contain the global semantic context are generated and then correlation maps derived from local and context-aware features are fused to apply the global semantic context to local features. A dynamical fusion mechanism [22] for fusing correlation maps to alleviate the performance degradation problem caused by background clutter has also been developed. However, this method requires a long execution time owing to its overuse of the 4D convolution kernels.…”

“…Since 4D convolution kernels are prohibitively expensive, these approaches are computationally intensive, resulting in lengthy execution time. Furthermore, their training method [22] is highly memory intensive since they require additional supervision from additional tasks to train the network.…”

“…We further compared its training memory usage and execution time with those of other neighborhood consensus (NC)-based methods. The results revealed that the proposed method is four times faster and uses one-third the training memory compared to the prior approaches [22], while producing competitive results relative to state-of-the-art weakly supervised semantic correspondence methods. We summarize our contributions as follows: first, our proposed global context fused feature representation makes our method faster than existing state-of-the-art methods; second, we introduce a semi-global self-similarity feature to efficiently mitigate the effect of background clutter on the captured global context; and third, we achieve competitive performance while reducing training memory usage compared to existing methods.…”