Kernel Discriminant Analysis for Positive Definite and Indefinite Kernels

Abstract

Kernel methods are a class of well established and successful algorithms for pattern analysis thanks to their mathematical elegance and good performance. Numerous nonlinear extensions of pattern recognition techniques have been proposed so far based on the so-called kernel trick. The objective of this paper is twofold. First, we derive an additional kernel tool that is still missing, namely kernel quadratic discriminant (KQD). We discuss different formulations of KQD based on the regularized kernel Mahalanobis distance in both complete and class-related subspaces. Secondly, we propose suitable extensions of kernel linear and quadratic discriminants to indefinite kernels. We provide classifiers that are applicable to kernels defined by any symmetric similarity measure. This is important in practice because problem-suited proximity measures often violate the requirement of positive definiteness. As in the traditional case, KQD can be advantageous for data with unequal class spreads in the kernel-induced spaces, which cannot be well separated by a linear discriminant. We illustrate this on artificial and real data for both positive definite and indefinite kernels.

One-Class Classification with Gaussian Processes

“…One approach to describe the data is to estimate a normal distribution in feature space H induced via a mapping Φ : X → H. It has been shown by Pȩkalska et al [13] that computing the variance term in GP regression is equal to the Mahalanobis distance (to the data mean in feature space) if the regularized (σ n > 0) kernel-induced scaling matrix Σ = Φ(X) Φ(X) T + σ 2 n I is used:…”

Subspace Learning in Krein Spaces: Complete Kernel Fisher Discriminant Analysis with Indefinite Kernels

“…Another family of useful npd kernels are the compact support (cs) kernels [16]. Popular nonEuclidean (nonmetric) similarities/dissimilarities, such as Hausdorff distances [17] and Kullback-Leibler divergence between probability distributions, can be used to define npd kernels [18,19]. Hence, there is both practical and theoretical need to properly handle all these measures and npd kernels in order to extract discriminant features using an KDA framework with npd kernels.…”

“…In particular in [21] a geometrical interpretation of learning a large margin classifier with indefinite kernels has been discussed. In [21] classification frameworks based on two-class Kernel Fisher Discriminant Analysis (KFDA) and in [18] Kernel Quadratic Discriminant (KQD) analysis with indefinite kernels were proposed. In this paper we study feature extraction with npd (or simple indefinite) kernels.…”

“…We would like to highlight that in [5,18,21] only classifiers based on quadratic discriminant functions and two class classifier based on IKFDA in Krein spaces were proposed. Our paper takes a different direction.…”

“…Summarizing the contributions of this paper are: (a) an Indefinite Kernel Principal Component Analysis in Krein Spaces 2 (b) a Complete Indefinite Kernel Fisher Discriminant Analysis (ICKFDA) in Krein spaces. We furthermore propose npd kernels that contrary to [18] achieve state-of-the-art performance in fully automatic face recognition.…”