Online learning for sparse PCA in high dimensions: Exact dynamics and phase transitions

Wang, Chuang; Lu, Yue M.

doi:10.1109/itw.2016.7606821

Cited by 6 publications

(2 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This work is an extension of a recent analysis on the dynamics of online sparse PCA [13] to more general settings. The idea of studying the scaling limits of online learning algorithms first appeared in a series of work that mostly came from the statistical physics communities [4,6,[14][15][16][17] in the 1990s.…”

Section: Introductionmentioning

confidence: 95%

The scaling limit of high-dimensional online independent component analysis

Wang¹,

Lu²

2019

J. Stat. Mech.

View full text Add to dashboard Cite

We analyze the dynamics of an online algorithm for independent component analysis in the high-dimensional scaling limit. As the ambient dimension tends to infinity, and with proper time scaling, we show that the time-varying joint empirical measure of the target feature vector and the estimates provided by the algorithm will converge weakly to a deterministic measured-valued process that can be characterized as the unique solution of a nonlinear PDE. Numerical solutions of this PDE, which involves two spatial variables and one time variable, can be efficiently obtained. These solutions provide detailed information about the performance of the ICA algorithm, as many practical performance metrics are functionals of the joint empirical measures. Numerical simulations show that our asymptotic analysis is accurate even for moderate dimensions. In addition to providing a tool for understanding the performance of the algorithm, our PDE analysis also provides useful insight. In particular, in the high-dimensional limit, the original coupled dynamics associated with the algorithm will be asymptotically "decoupled", with each coordinate independently solving a 1-D effective minimization problem via stochastic gradient descent. Exploiting this insight to design new algorithms for achieving optimal trade-offs between computational and statistical efficiency may prove an interesting line of future research.

show abstract

Section: Introductionmentioning

confidence: 95%

The scaling limit of high-dimensional online independent component analysis

Wang¹,

Lu²

2019

J. Stat. Mech.

View full text Add to dashboard Cite

show abstract

“…For instance, in deep learning, stochastic gradient descent is the most popular training algorithm [12]. From the theoretical point of view, the restriction to the fully online case, where a single data point is used at a time, offers interesting possibilities of analysis, as demonstrated in different machine learning problems by [13][14][15]. Here we will consider the Bayesian online learning of the calibration variables.…”

Section: Introductionmentioning

confidence: 99%

Blind calibration for compressed sensing: state evolution and an online algorithm

Gabrié¹,

Barbier²,

Krząkała³

et al. 2020

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

Compressed sensing, allows to acquire compressible signals with a small number of measurements. In applications, a hardware implementation often requires a calibration as the sensing process is not perfectly known. Blind calibration, that is performing at the same time calibration and compressed sensing is thus particularly appealing. A potential approach was suggested by Schülke and collaborators in [1, 2], using approximate message passing (AMP) for blind calibration (cal-AMP). Here, the algorithm is extended from the already proposed offline case to the online case, where the calibration is refined step by step as new measured samples are received. Furthermore, we show that the performance of both the offline and the online algorithms can be theoretically studied via the State Evolution (SE) formalism. Through numerical simulations, the efficiency of cal-AMP and the consistency of the theoretical predictions are confirmed.

show abstract

Gradient-based sparse principal component analysis with extensions to online learning

2022

View full text Add to dashboard Cite

Summary Sparse principal component analysis (PCA) is an important technique for simultaneous dimensionality reduction and variable selection of high-dimensional data. In this work we combine the unique geometric structures of the sparse PCA problem with recent advances in convex optimization to develop novel gradient-based sparse PCA algorithms. These new algorithms enjoy the same global convergence guarantee as the original alternating direction method of multipliers, and can be more efficiently implemented with the rich toolbox developed for gradient methods in the deep learning literature. Most notably, these gradient-based algorithms can be combined with stochastic gradient descent methods, leading to efficient online sparse PCA algorithms with provable numerical and statistical performance guarantees. The practical performance and usefulness of the new algorithms are demonstrated in various simulation studies. As an application, the scalability and statistical accuracy of our method enable us to find interesting functional gene groups in high-dimensional RNA sequencing data.

show abstract

Online learning for sparse PCA in high dimensions: Exact dynamics and phase transitions

Cited by 6 publications

References 16 publications

The scaling limit of high-dimensional online independent component analysis

The scaling limit of high-dimensional online independent component analysis

Blind calibration for compressed sensing: state evolution and an online algorithm

Gradient-based sparse principal component analysis with extensions to online learning

Contact Info

Product

Resources

About