Generalized Thresholding of Large Covariance Matrices

Abstract

We propose a new class of generalized thresholding operators that combine thresholding with shrinkage, and study generalized thresholding of the sample covariance matrix in high dimensions. Generalized thresholding of the covariance matrix has good theoretical properties and carries almost no computational burden. We obtain an explicit convergence rate in the operator norm that shows the tradeoff between the sparsity of the true model, dimension, and the sample size, and shows that generalized thresholding is consistent over a large class of models as long as the dimension p and the sample size n satisfy log p/n ! 0. In addition, we show that generalized thresholding has the ''sparsistency'' property, meaning it estimates true zeros as zeros with probability tending to 1, and, under an additional mild condition, is sign consistent for nonzero elements. We show that generalized thresholding covers, as special cases, hard and soft thresholding, smoothly clipped absolute deviation, and adaptive lasso, and compare different types of generalized thresholding in a simulation study and in an example of gene clustering from a microarray experiment with tumor tissues.

Design-free estimation of variance matrices

“…We consider the shrinkage towards identity or equicorrelation estimators proposed by Ledoit and Wolf (2004a,b), setting the shrinkage parameters to their respective optimal values derived in the aforementioned papers. We also consider the hard and soft thresholding estimators due to Bickel and Levina (2008b) and Rothman, Levina, and Zhu (2009). The various fine-tuning parameters required to calculate these estimators are set in accordance with the default values of the R code kindly shared with us by Adam J. Rothman.…”

Positive-Definite ℓ1-Penalized Estimation of Large Covariance Matrices

“…The first dataset has 64 training tissue samples with four types of tumors (23 EWS, 8 BL-NHL, 12 NB, and 21 RMS) and 6567 gene expression values for each sample. We applied the prefiltering step used in Khan et al (2001) and then picked the top 40 and bottom 160 genes based on the F-statistic as done in Rothman, Levina, and Zhu (2009). The second dataset has 63 subjects with 44 healthy controls and 19 cardiovascular patients, and 20,426 genes measured for each subject.…”

“…The usual sample covariance matrix is optimal in the classical setting with large samples and fixed low dimensions (Anderson 1984), but it performs very poorly in the highdimensional setting (Marčenko and Pastur 1967;Johnstone 2001). In the recent literature, regularization techniques have been used to improve the sample covariance matrix estimator, including banding (Wu and Pourahmadi 2003;Bickel and Levina 2008a), tapering (Furrer and Bengtsson 2007;Cai, Zhang, and Zhou 2010), and thresholding (Bickel and Levina 2008b;El Karoui 2008;Rothman, Levina, and Zhu 2009). Banding or tapering is very useful when the variables have a natural ordering and off-diagonal entries of the target covariance matrix decay to zero as they move away from the diagonal.…”

“…Rothman, Levina, and Zhu (2009) where s λ (z) is the generalized thresholding function. The generalized thresholding function covers a number of commonly used shrinkage procedures, for example, the hard thresholding s λ (z) = zI {|z|>λ} , the soft thresholding s λ (z) = sign(z)(|z| − λ) + , the smoothly clipped absolute deviation thresholding (Fan and Li 2001), and the adaptive lasso thresholding (Zou 2006).…”

“…The generalized thresholding function covers a number of commonly used shrinkage procedures, for example, the hard thresholding s λ (z) = zI {|z|>λ} , the soft thresholding s λ (z) = sign(z)(|z| − λ) + , the smoothly clipped absolute deviation thresholding (Fan and Li 2001), and the adaptive lasso thresholding (Zou 2006). Consistency results and explicit rates of convergence have been obtained for these regularized estimators in the literature, for example, Bickel and Levina (2008a, b), El Karoui (2008), Rothman, Levina, and Zhu (2009), and Cai and Liu (2011). The recent articles by Cai and Zhou (2012a, b) have established the minimax optimality of the thresholding estimator for estimating a wide range of large sparse covariance matrices under commonly used matrix norms.…”