A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis

Abstract

We present a penalized matrix decomposition (PMD), a new framework for computing a rank-K approximation for a matrix. We approximate the matrix X as circumflexX = sigma(k=1)(K) d(k)u(k)v(k)(T), where d(k), u(k), and v(k) minimize the squared Frobenius norm of X - circumflexX, subject to penalties on u(k) and v(k). This results in a regularized version of the singular value decomposition. Of particular interest is the use of L(1)-penalties on u(k) and v(k), which yields a decomposition of X using sparse vectors. We show that when the PMD is applied using an L(1)-penalty on v(k) but not on u(k), a method for sparse principal components results. In fact, this yields an efficient algorithm for the "SCoTLASS" proposal (Jolliffe and others 2003) for obtaining sparse principal components. This method is demonstrated on a publicly available gene expression data set. We also establish connections between the SCoTLASS method for sparse principal component analysis and the method of Zou and others (2006). In addition, we show that when the PMD is applied to a cross-products matrix, it results in a method for penalized canonical correlation analysis (CCA). We apply this penalized CCA method to simulated data and to a genomic data set consisting of gene expression and DNA copy number measurements on the same set of samples.

Generalized Low Rank Models

“…Singh and Gordon [SG08] offered a complete view of the state of the literature on matrix factorization in Table 1 of their 2008 paper, and noted that by changing the loss function and regularizer, one may recover algorithms including PCA, weighted PCA, k-means, k-medians, 1 SVD, probabilistic latent semantic indexing (pLSI), nonnegative matrix factorization with 2 or KL-divergence loss, exponential family PCA, and MMMF. Witten et al introduced the statistics community to sparsity-inducing matrix factorization in a 2009 paper on penalized matrix decomposition, with applications to sparse PCA and canonical correlation analysis [WTH09]. Recently, Markovsky's monograph on low rank approximation [Mar12] reviewed some of this literature, with a focus on applications in system, control, and signal processing.…”

“…The earliest example of this approach is canonical correlation analysis, developed by Hotelling [Hot36] in 1936 to understand the relations between two sets of variates in terms of the eigenvectors of their covariance matrix. This approach was extended by Witten et al [WTH09] to encourage structured (e.g., sparse) factors. In the 1970s, De Leeuw et al proposed the use of low rank models to fit data measured in nominal, ordinal and cardinal levels [DLYT76].…”

Robust groupwise least angle regression

“…Literature on other applications of high-dimensional data analysis shows that imposing a diagonal covariance or correlation matrix can lead to good results (e.g. Dudoit et al, 2001;Tibshirani et al, 2003;Witten et al, 2009). In this case, the data cleaning weights ω = (ω 1 , .…”

Probabilistic penalized principal component analysis

“…Zou et al (2006) exploit the regression/reconstruction error property of principal components in order to obtain sparse principal components. Witten et al (2009) proposed a penalized matrix decomposition with L 1 penalty that results in a regularized version of singular value decomposition for sparse principal components and canonical correlation analysis. These are able to enhance computing efficiency applicability from principal components to other methods; however, still we need more advanced algorithms when we decide to use other non-convex penalties like hard penalties or smoothly clipped absolute deviation (SCAD) rather than least absolute shrinkage and selection operator (LASSO) type L 1 penalty function.…”