Joint geometric graph embedding for partial shape matching in images

Abstract

A novel multi-criteria optimization framework for matching of partially visible shapes in multiple images using joint geometric graph embedding is proposed. The proposed framework achieves matching of partial shapes in images that exhibit extreme variations in scale, orientation, viewpoint and illumination and also instances of occlusion; conditions which render impractical the use of global contour-based descriptors or local pixel-level features for shape matching. The proposed technique is based on optimization of the embedding distances of geometric features obtained from the eigenspectrum of the joint image graph, coupled with regularization over values of the mean pixel intensity or histogram of oriented gradients. It is shown to obtain successfully the correspondences denoting partial shape similarities as well as correspondences between feature points in the images. A new benchmark dataset is proposed which contains disparate image pairs with extremely challenging variations in viewing conditions when compared to an existing dataset [18]. The proposed technique is shown to significantly outperform several state-ofthe-art partial shape matching techniques on both datasets.

Matching Disparate Image Pairs Using Shape-Aware ConvNets

“…Both, local point-based and global shape-based image matching approaches have their advantages and shortcomings. While global shape-based descriptors are well behaved and less sensitive to outliers, they, in isolation, are insufficient to compute point-wise correspondences since they do not explicitly encode keypoint information as appearance-based descriptors do [2,6]. Although global shape descriptors are shown to perform well when the image pairs exhibit significant shape deformations due to changes in viewpoint, their performance suffers in the presence of strong shape articulations [6].…”

“…While there have been several attempts to combine local and global cues for image matching, we draw our inspiration from [2,6] which show that a higherlevel shape representation computed via eigenspectral analysis of a low-level feature representation, can yield matching features that are persistent across illumination changes and other affine variations [2]. In particular, features that encode the extrema of the eigenfunctions of the joint image graph are shown to be stable, persistent and robust across wide range of illumination variations [2,6]. While the proposed hybrid approach combines both global and local cues efficiently, a key question remains, i.e., how does one extend the approach to enable end-to-end learning in a data-driven manner?…”

“…The top-m eigenvectors computed in Section 4.2, are fed to a cascade of convolutional and dense layers followed by max pooling layers to induce non-linearity ( Figure 2). Only the non-trivial or top-m eigenvectors are considered since they encode rich global shape information whereas eigenvectors that are trivial or outside the top-m preserve mostly local cues which we deem uninformative and noisy [6] and hence detrimental to the overall learning. Finally, the network outputs 9 values, which are then reshaped into a 3 × 3 grid that constitutes the homography matrix denoted as H. The use of the homography matrix has an added advantage as they project the points to target in a regularized manner.…”

“…There have been some recent attempts at exploiting high-level object shape representations in image matching [2,6]. However, only a few attempts have focused on deriving more meaningful low-level representations of the underlying data rather than on matching patches or regions across images [2,6]. In this paper, we propose an novel CNN architecture, trained in an endto-end fashion, that attempts to determine a global shape-based representation of scene structures from local features.…”

“…The eigenvectors are used to learn to regress the homography that minimizes a point-wise correspondence loss function and to train the network to learn to reason about image matching via the computed shape representation. While the eigenvalues and eigenvectors obtained via eigenspectral decomposition of the affinity matrix are shown to reliably represent the underlying global object shape [6], reasoning about object shapes purely as linear combinations of the eigenvectors (using the eigenvalues as coefficients) is insufficient to capture the complexities of real-world shapes, especially in the face of large image disparities. The goal of this work is to relax the linearity constraints and introduce nonlinearity within the shape eigenspace using CNNs.…”