Encoding Classes of Unaligned Objects Using Structural Similarity Cross-Covariance Tensors

Abstract

Encoding an object essence in terms of self-similarities between its parts is becoming a popular strategy in Computer Vision. In this paper, a new similarity-based descriptor, dubbed Structural Similarity Cross-Covariance Tensor is proposed, aimed to encode relations among different regions of an image in terms of cross-covariance matrices. The latter are calculated between low-level feature vectors extracted from pairs of regions. The new descriptor retains the advantages of the widely used covariance matrix descriptors [1], extending their expressiveness from local similarities inside a region to structural similarities across multiple regions. The new descriptor, applied on top of HOG, is tested on object and scene classification tasks with three datasets. The proposed method always outclasses baseline HOG and yields significant improvement over a recently proposed self-similarity descriptor in the two most challenging datasets.

Encoding Structural Similarity by Cross-Covariance Tensors for Image Classification

“…The present work considerably extends the study presented in 12 , by detailing how the SS-CCT can be quickly computed through integral images 5 , and showing numerical experiments. We also revise the experimental protocol for the scene recognition, obtaining results that are more generalizable.…”

“…Assuming to sort the regions in a row wise manner, i.e. the first N h regions have ∆X Ri = 0, it holds that ∆Y Rj ≥ ∆Y Ri for each j ≥ i and consequently ∆Y ≥ 0, allowing to reduce the number of possible relative displacements to (2N h − 1)N v , as previously defined in (12). Overall the computation of Eq.…”

Kernelized covariance for action recognition

“…The complexity of our trial-specific kernelized covariance is O(M 2 T 2 ). Thus, differently from previous approaches [27], [33], [28], [29], the proposed framework is very efficient if compared to the cubic complexity of methods like [33] which require eigen-decomposition. Under a mathematical point of view, our kernelized covariance is a natural generalization of the classical covariance matrix, which can be retrieved as a particular case in our paradigm once fixed the kernel function (9) to be a linear one.…”

“…The usage of the covariance S to produce descriptors for classification tasks has been intensively studied [23], [24], [25], [26], [27], [28], [29], [17]. In particular, [23] proposed patch-specific covariance descriptors, efficiently computed with integral images.…”

Optimizing Features Quality: A Normalized Covariance Fusion Framework for Skeleton Action Recognition

“…This latter direction grounds on the mathematical properties of positive definite matrices, exploiting the Riemannian metrics on the manifold for image classification. Once moved from a finite to an infinitedimensional space, the performance enhances [40], [41] and only recently, the deep learning approaches have shown to be superior. However, one of the main limitation related to the covariance matrix is that it only enables to capture the linear inter-relationships [42].…”