Sparse PCA: Optimal rates and adaptive estimation

Abstract

Principal component analysis (PCA) is one of the most commonly used statistical procedures with a wide range of applications. This paper considers both minimax and adaptive estimation of the principal subspace in the high dimensional setting. Under mild technical conditions, we first establish the optimal rates of convergence for estimating the principal subspace which are sharp with respect to all the parameters, thus providing a complete characterization of the difficulty of the estimation problem in term of the convergence rate. The lower bound is obtained by calculating the local metric entropy and an application of Fano's lemma. The rate optimal estimator is constructed using aggregation, which, however, might not be computationally feasible. We then introduce an adaptive procedure for estimating the principal subspace which is fully data driven and can be computed efficiently. It is shown that the estimator attains the optimal rates of convergence simultaneously over a large collection of the parameter spaces. A key idea in our construction is a reduction scheme which reduces the sparse PCA problem to a high-dimensional multivariate regression problem. This method is potentially also useful for other related problems.Comment: Published in at http://dx.doi.org/10.1214/13-AOS1178 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

Projected Principal Component Analysis in Factor Models

“…The separations between the common factors and idiosyncratic components are carried out by the low-rank plus sparsity decomposition. See, for example, Cai et al (2013); Candès and Recht (2009) ;Fan et al (2013); Koltchinskii et al (2011);Ma (2013); Negahban and Wainwright (2011).…”

A Literature Survey on High-Dimensional Sparse Principal Component Analysis

“…The main questions needed to be answered in sparse PCA is whether there has an algorithm not only asymptotically consistent but also computationally efficient. Theoretical research from statistical guarantees view of sparse PCA includes consistency [2,8,14,38,41,50,53,55], minimax risk bounds for estimating eigenvectors [40,[42][43]45,61], optimal sparsity level detection [4,44,48,59] and principal subspaces estimation [5,[15][16]36,9,40,51,57] have been established under various statistical models. Because most of the methods based on spiked covariance model, so we firstly given an introduction about spiked variance model and then give a high dimensional sparse PCA theoretical analysis review from above several aspects.…”

“…Ma et al [15] presented a iterative thresholding method and proved its consistency and achieved a near optimal statistical convergence rates when estimating several individual leading vectors under the spiked covariance model with the similar condition in [42]. Cai et al [43,45] attained an optimal principal subspace estimator based on a regression-type method, and the minimax rates of convergence are derived and a computationally efficient adaptive estimator is constructed. Vu et al [16] proposed a new method called FPS which generalized DSPCA to estimate the principal subspace spanned by the top k leading eigenvectors.…”

Large covariance estimation through elliptical factor models

“…After that, substantial amount of work has focused on the inference of high-dimensional covariance matrices under unconditional sparsity, that is, the covariance matrix itself is sparse (Cai and Liu, 2011; Cai, Ren and Zhou, 2013; Cai, Zhang and Zhou, 2010; Karoui, 2008; Lam and Fan, 2009; Ravikumar et al, 2011) or conditional sparsity, that is, the covariance matrix is sparse after subtraction by a low-rank component (Amini and Wainwright, 2008; Berthet and Rigollet, 2013a,b; Birnbaum et al, 2013; Cai, Ma and Wu, 2013, 2015; Johnstone and Lu, 2009; Levina and Vershynin, 2012; Rothman, Levina and Zhu, 2009; Ma, 2013; Shen, Shen and Marron, 2013; Paul and Johnstone, 2012; Vu and Lei, 2013; Zou, Hastie and Tibshirani, 2006). This research area is very active, and as a result, this list of references is illustrative rather than comprehensive.…”

“…In the current paper, we further explain the benefit of pervasive structure in a more transparent way, along with a weaker sub-Gaussian assumption (see discussions after Theorem 3.2). A surprising result is that the diverging signal of spiked eigenvalues excludes the necessity of the sparse principal component assumption in sparse PCA literature, comparing with for example Cai, Ma and Wu (2013).…”