Unsupervised and Supervised Principal Component Analysis: Tutorial

Abstract

This is a detailed tutorial paper which explains the Principal Component Analysis (PCA), Supervised PCA (SPCA), kernel PCA, and kernel SPCA. We start with projection, PCA with eigendecomposition, PCA with one and multiple projection directions, properties of the projection matrix, reconstruction error minimization, and we connect to auto-encoder. Then, PCA with singular value decomposition, dual PCA, and kernel PCA are covered. SPCA using both scoring and Hilbert-Schmidt independence criterion are explained. Kernel SPCA using both direct and dual approaches are then introduced. We cover all cases of projection and reconstruction of training and out-of-sample data. Finally, some simulations are provided on Frey and AT&T face datasets for verifying the theory in practice.

Spectral, Probabilistic, and Deep Metric Learning: Tutorial and Survey

“…Relevant Component Analysis (RCA) (Shental et al, 2002) is a metric learning method. In this method, we first apply Principal Component Analysis (PCA) (Ghojogh & Crowley, 2019) on data using the total scatter of data. Let the projection matrix of PCA be denoted by U .…”

“…Inspired by Fisher discriminant analysis (Fisher, 1936;Ghojogh et al, 2019b), we maximize the inter-class variances of projected data, v r=1 tr(U S (r) b U ), to discriminate the classes after projection. Also, inspired by principal component analysis (Ghojogh & Crowley, 2019), we maximize the total scatter of projected data, v r=1 tr(U S (r) t U ), for expressiveness. Moreover, we maximize the dependence of the projected data in all views because various views of a point should be related.…”

“…The code layer between the encoder and decoder is the embedding space of metric. Note that if the activation functions of all layers are linear, the under-complete autoencoder is reduced to Principal Component Analysis (Ghojogh & Crowley, 2019). Let U l denote the weight matrix of the l-th layer of network, e be the number of layers of encoder, and d be the number of layers of decoder.…”

Laplacian-Based Dimensionality Reduction Including Spectral Clustering, Laplacian Eigenmap, Locality Preserving Projection, Graph Embedding, and Diffusion Map: Tutorial and Survey

“…Convergence analysis of Laplacian eigenmap methods can be found in several papers (Belkin & Niyogi, 2006;Singer, 2006). Inspired by eigenfaces (Turk & Pentland, 1991) and Fisherfaces (Belhumeur et al, 1997) which have been proposed based on principal component analysis (Ghojogh & Crowley, 2019) and Fisher discriminant analysis (Ghojogh et al, 2019b), respectively, Laplacianfaces (He et al, 2005) has been proposed for face recognition using Laplacian eigenmaps. Finally, a recent survey on Laplacian eigenmap is (Li et al, 2019).…”

PCA-based Multi Task Learning: a Random Matrix Approach

“…A series of supervised (single-task) learning methods were proposed which rely on PCA [7,41,53,17]: the central idea is to project the available data onto a shared low-dimensional space, thus ignoring individual data variations. These algorithms are generically coined supervised principal component analysis (SPCA).…”