Neural Network Implementations for PCA and Its Extensions

Abstract

Many information processing problems can be transformed into some form of eigenvalue or singular value problems. Eigenvalue decomposition (EVD) and singular value decomposition (SVD) are usually used for solving these problems. In this paper, we give an introduction to various neural network implementations and algorithms for principal component analysis (PCA) and its various extensions. PCA is a statistical method that is directly related to EVD and SVD. Minor component analysis (MCA) is a variant of PCA, which is useful for solving total least squares (TLSs) problems. The algorithms are typical unsupervised learning methods. Some other neural network models for feature extraction, such as localized methods, complex-domain methods, generalized EVD, and SVD, are also described. Topics associated with PCA, such as independent component analysis (ICA) and linear discriminant analysis (LDA), are mentioned in passing in the conclusion. These methods are useful in adaptive signal processing, blind signal separation (BSS), pattern recognition, and information compression.

Unsupervised learning and clustered connectivity enhance reinforcement learning in spiking neural networks

“…A more plausible approach is based on “competitive learning” as described in Hertz et al (1991), which employs Oja’s rule (modified Hebbian learning with multiplicative normalization). This model was shown to perform PCA, constituting a powerful, biologically-plausible alternative for feature extraction and dimensionality reduction (Oja, 1982), see also Qiu et al (2012) for a detailed review on neural networks implementing PCA or non-linear extensions of PCA. In a recent study, competitive unsupervised learning applied to the lower layers of a feedforward neural network was shown to successfully lead to good generalization performance on the MNIST dataset (Krotov & Hopfield, 2019).…”

Biologically Plausible Online Principal Component Analysis Without Recurrent Neural Dynamics

“…If we take Λ = I, the dynamical system (8) reduces to that of Pehlevan, Sengupta, and Chklovskii [3, Theorem 1]. They proved that, at any stable fixed point (M FP , W FP ), M −1 FP W FP has orthogonal rows spanning the principal subspace of X.…”

“…The dynamics (8) and (9) seem to require the matrix inverse M −1 (s) which is cumbersome for a biologically plausible neural network and required iterations in the previous similarity matching networks. However, Theorems 1 and 2 show that at fixed points M FP is diagonal, which forms the basis of our approach for iteration-free dynamics.…”

“…At a fixed point, we have from(8) that F FP C = W FP and F FP CF FP = ΛM FP Λ. Together these imply W FP F FP = ΛM FP Λ, but by definition F FP ≡ M −1 FP W FP so this gives W FP W FP M −1 FP = ΛM FP Λ and thus ΛM FP ΛM FP = M FP ΛM FP Λ.…”

Data Mining in Customer Profitability Analysis

“…forward direction links), consisting of three layers; the input, hidden and output layers [15]. This kind of networks is called the autoassociative network [16]. The performance of a MLP depends on its generalization capability, meaning the data representation.…”