Fisher and Kernel Fisher Discriminant Analysis: Tutorial

Abstract

This is a detailed tutorial paper which explains the Fisher discriminant Analysis (FDA) and kernel FDA. We start with projection and reconstruction. Then, one-and multi-dimensional FDA subspaces are covered. Scatters in two-and then multi-classes are explained in FDA. Then, we discuss on the rank of the scatters and the dimensionality of the subspace. A real-life example is also provided for interpreting FDA. Then, possible singularity of the scatter is discussed to introduce robust FDA. PCA and FDA directions are also compared. We also prove that FDA and linear discriminant analysis are equivalent. Fisher forest is also introduced as an ensemble of fisher subspaces useful for handling data with different features and dimensionality. Afterwards, kernel FDA is explained for both one-and multi-dimensional subspaces with both two-and multi-classes. Finally, some simulations are performed on AT&T face dataset to illustrate FDA and compare it with PCA.

Spectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey

“…If the class labels are available, metric learning tries to make the intra-class and inter-class variances smaller and larger, respectively. This is the same idea as the idea of Fisher Discriminant Analysis (FDA) (Fisher, 1936;Ghojogh et al, 2019b).…”

“…ANALYSIS Another metric learning method is (Alipanahi et al, 2008) which has two approaches, introduced in the following. The relation of metric learning with Fisher discriminant analysis (Fisher, 1936;Ghojogh et al, 2019b) was discussed in this paper (Alipanahi et al, 2008).…”

“…The optimization problem is similar to the optimization of Fisher discriminant analysis (FDA) (Fisher, 1936;Ghojogh et al, 2019b) where Σ S and Σ D are replaced with the intraclass and inter-class covariance matrices of data, respectively. This shows the relation of this method with FDA.…”

“…According to Proposition 2, metric learning can be seen as Euclidean distance after projection onto the column space of a projection matrix U where W = U U . Similar to Fisher discriminant analysis (Fisher, 1936;Ghojogh et al, 2019b), DCA maximizes the inter-class variance and minimizes the intra-class variance after projection. Hence, its optimization is:…”

“…The total scatter of data, in the r-th view, is denoted by S (r) t and is the covariance of data in that view. Inspired by Fisher discriminant analysis (Fisher, 1936;Ghojogh et al, 2019b), we maximize the inter-class variances of projected data, v r=1 tr(U S (r) b U ), to discriminate the classes after projection. Also, inspired by principal component analysis (Ghojogh & Crowley, 2019), we maximize the total scatter of projected data, v r=1 tr(U S (r) t U ), for expressiveness.…”