On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix

Oja, Erkki; Karhunen, Juha

doi:10.1016/0022-247x(85)90131-3

Cited by 472 publications

(332 citation statements)

References 7 publications

Supporting

Mentioning

326

Contrasting

Unclassified

Order By: Relevance

“…As mentioned earlier, the classical Oja's method [16] can be viewed as a special case of the algorithm in (5). It corresponds to setting φ(x) = 0 in (6), i.e., the algorithm does not apply the nonlinear mapping η(x).…”

Section: B the Nonsparse Case: Oja's Methodsmentioning

confidence: 99%

“…The update steps in (5) as well as the expression in (6) need some explanations. First, we note that, without the nonlinear mapping (i.e., by setting η(x) = x), the recursions in (5) are exactly the original Oja's method [16] for online PCA. The nonlinearity (6) in η(·) is introduced to promote sparsity of the estimates.…”

Section: Online Algorithm For Sparse Pcamentioning

confidence: 99%

“…In this paper, we analyze an online sparse PCA algorithm that combines the classical Oja's method [16] with an element-wise nonlinearity (e.g., soft-thresholding) at each iteration to promote sparsity (see Section II for the exact form.) Specifically, let x k be the estimate of ξ given by the algorithm upon receiving the kth sample; let x i k and ξ i denote the ith component of each vector.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Online learning for sparse PCA in high dimensions: Exact dynamics and phase transitions

Wang

2016

2016 IEEE Information Theory Workshop (ITW)

View full text Add to dashboard Cite

Abstract-We study the dynamics of an online algorithm for learning a sparse leading eigenvector from samples generated from a spiked covariance model. This algorithm combines the classical Oja's method for online PCA with an element-wise nonlinearity at each iteration to promote sparsity. In the high-dimensional limit, the joint empirical measure of the underlying sparse eigenvector and its estimate provided by the algorithm is shown to converge weakly to a deterministic, measure-valued process. This scaling limit is characterized as the unique solution of a nonlinear PDE, and it provides exact information regarding the asymptotic performance of the algorithm. For example, performance metrics such as the cosine similarity and the misclassification rate in sparse support recovery can be obtained by examining the limiting dynamics. A steady-state analysis of the nonlinear PDE also reveals an interesting phase transition phenomenon. Although our analysis is asymptotic in nature, numerical simulations show that the theoretical predictions are accurate for moderate signal dimensions.

show abstract

Section: B the Nonsparse Case: Oja's Methodsmentioning

confidence: 99%

Section: Online Algorithm For Sparse Pcamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Online learning for sparse PCA in high dimensions: Exact dynamics and phase transitions

Wang

2016

2016 IEEE Information Theory Workshop (ITW)

View full text Add to dashboard Cite

show abstract

“…where R xx is a symmetric positive-definite matrix, then (5) and (6) forms the framework for principal subspace analysis (PSA) or minor subspace analysis (MSA) [23][24][25][26][27][28][29][30][31][32]. Alternatively, if J (W) is a contrast function, then (5) and (6) forms the framework for contrast-based blind source separation of prewhitened instantaneous mixtures [33][34][35][36].…”

Section: Grassmann and Stiefel Manifoldsmentioning

confidence: 99%

“…Although other adaptive methods could be used, gradient methods represent the simplest to implement and thus form the baseline to which others are often compared. Our algorithms are spatio-temporal extensions of gradient techniques on the Grassmann and Stiefel manifolds [21][22][23][24][25][26][27][28][29][30] and can be applied to any appropriatelydefined cost function. We prove that our algorithms in differential form preserve (2) or (4) over time, and thus they adjust W p in the impulse response space of paraunitary systems.…”

Section: Introductionmentioning

confidence: 99%

Gradient Adaptive Paraunitary Filter Banks for Spatio-Temporal Subspace Analysis and Multichannel Blind Deconvolution

Douglas

Амари

Kung

2004

The Journal of VLSI Signal Processing-Systems for Signal, Image

View full text Add to dashboard Cite

Abstract. Paraunitary filter banks are important for several signal processing tasks, including coding, multichannel deconvolution and equalization, adaptive beamforming, and subspace processing. In this paper, we consider the task of adapting the impulse response of a multichannel paraunitary filter bank via gradient ascent or descent on a chosen cost function. Our methods are spatio-temporal generalizations of gradient techniques on the Grassmann and Stiefel manifolds, and we prove that they inherently maintain the paraunitariness of the multichannel adaptive system over time. We then discuss the necessary practical approximations, modifications, and simplifications of the methods for solving two relevant signal processing tasks: (i) spatio-temporal subspace analysis and (ii) multichannel blind deconvolution. Simulations indicate that our methods can provide simple, useful solutions to these important problems.

show abstract

Practical Considerations

2001

Independent Component Analysis

View full text Add to dashboard Cite

3.4 Concluding remarks and references Problems 4 Estimation Theory 4.1 Basic concepts 4.2 Properties of estimators 4.3 Method of moments 4.4 Least-squares estimation 4.4.1 Linear least-squares method 4.4.2 Nonlinear and generalized least squares * 4.5 Maximum likelihood method 4.6 Bayesian estimation * 4.6.1 Minimum mean-square error estimator 4.6.2 Wiener filtering 4.6.3 Maximum a posteriori (MAP) estimator 4.7 Concluding remarks and references Problems

show abstract

On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix

Cited by 472 publications

References 7 publications

Online learning for sparse PCA in high dimensions: Exact dynamics and phase transitions

Online learning for sparse PCA in high dimensions: Exact dynamics and phase transitions

Gradient Adaptive Paraunitary Filter Banks for Spatio-Temporal Subspace Analysis and Multichannel Blind Deconvolution

Practical Considerations

Contact Info

Product

Resources

About