Algorithms and networks for accelerated convergence of adaptive LDA

Abstract

We introduce and discuss new accelerated algorithms for linear discriminant analysis (LDA) in unimodal multiclass Gaussian data. These algorithms use a variable step size, optimally computed in each iteration using (i) the steepest descent, (ii) conjugate direction, and (iii) Newton-Raphson methods in order to accelerate the convergence of the algorithm. Current adaptive methods based on the gradient descent optimization technique use a fixed or a monotonically decreasing step size in each iteration, which results in a slow convergence rate. Furthermore, the convergence of these algorithms depends on appropriate choices of the step sizes. The new algorithms have the advantage of automatic optimal selection of the step size using the current data samples. Based on the new adaptive algorithms, we present self-organizing neural networks for adaptive computation of −1/2 and use them in cascaded form with a PCA network for LDA. Experimental results demonstrate fast convergence and robustness of the new algorithms and justify their advantages for on-line pattern recognition applications with stationary and non-stationary multidimensional input data.

Computational Intelligence-Based Biometric Technologies

“…Chatterjee and Roychowdhury further proposed to use a Q −1/2 algorithm to train the first layer and an adaptive eigenvector computation algorithm to train the second layer [34]. Subsequently Moghaddam and Zadeh introduced a modified version of a self-organizing neural network by using the steepest descent optimization method to compute Q −1/2 , and thus accelerate the convergence of adaptive LDA [35], [36].…”

Fast adaptive LDA using quasi-Newton algorithm

“…Adaptive linear discriminant analysis (LDA) has already been introduced for dimensionality reduction in on-line pattern recognition applications (Mao and Jain, 1995;Chatterjee and Roychowdhury, 1997;Abrishami Moghaddam and Amiri-Zadeh, 2003;Abrishami Moghaddam et al, 2005). In (Mao and Jain, 1995), the authors proposed a two-layer network for LDA, each of which was a principal component analysis (PCA) network.…”

“…In (Abrishami Moghaddam and Amiri-Zadeh, 2003), the authors derived a fast adaptive algorithm for LDA based on the steepest descent optimization method. In a recent work (Abrishami Moghaddam et al, 2005), they proposed two new accelerated convergence adaptive LDA algorithms using conjugate direction and Newton-Raphson methods. In the adaptive Newton-Raphson LDA algorithm, a direct calculation of the inverse Hessian matrix has been proposed by the authors, which can be both laborious to calculate and invert for systems with large number of dimensions.…”

“…In the adaptive Newton-Raphson LDA algorithm, a direct calculation of the inverse Hessian matrix has been proposed by the authors, which can be both laborious to calculate and invert for systems with large number of dimensions. This paper proposes an improvement to the adaptive Newton-Raphson LDA algorithm presented in previous work (Abrishami Moghaddam et al, 2005). In the present work, we use the secant method for adaptive estimation of the inverse Hessian Matrix.…”

“…Recently, accelerated adaptive algorithms for R À1/2 computation based on the steepest descent, conjugate direction and Newton-Raphson methods and modified networks for fast adaptive LDA have been presented (Abrishami Moghaddam and Amiri-Zadeh, 2003;Abrishami Moghaddam et al, 2005).…”