Probabilistic penalized principal component analysis

Abstract

A variable selection method based on probabilistic principal component analysis (PCA) using penalized likelihood method is proposed. The proposed method is a two-step variable reduction method. The first step is based on the probabilistic principal component idea to identify principle components. The penalty function is used to identify important variables in each component. We then build a model on the original data space instead of building on the rotated data space through latent variables (principal components) because the proposed method achieves the goal of dimension reduction through identifying important observed variables. Consequently, the proposed method is of more practical use. The proposed estimators perform as the oracle procedure and are root-n consistent with a proper choice of regularization parameters. The proposed method can be successfully applied to high-dimensional PCA problems with a relatively large portion of irrelevant variables included in the data set. It is straightforward to extend our likelihood method in handling problems with missing observations using EM algorithms. Further, it could be effectively applied in cases where some data vectors exhibit one or more missing values at random.

On some limitations of probabilistic models for dimension‐reduction: Illustration in the case of probabilistic formulations of partial least squares

“…The estimation can deal with missing data, while still being computationally efficient (Tipping & Bishop, 1999; Zheng et al, 2016). Moreover, covariates can be included in the model (Chiquet, Mariadassou, & Robin, 2017), and penalized versions of the likelihood can be used to encourage sparsity or structured sparsity, for example, in a high‐dimensional framework (Guan & Dy, 2009; Park, Wang, & Mo, 2017; Zeng, Liu, Huang, & Liang, 2017). Finally, the probabilistic formulation is very versatile, and turns several complex settings into natural extensions of the simple Gaussian ones mentioned above.…”

On some limitations of probabilistic models for dimension-reduction: Illustration in the case of probabilistic formulations of partial least squares

“…The estimation can be computationally more efficient, and can deal with missing data (Tipping andBishop, 1999, Zheng et al, 2016). Moreover, covariates can be included in the model (Chiquet et al, 2017), and penalized versions of the likelihood can be used to encourage sparsity or structured sparsity, in particular in a high-dimensional framework (Guan and Dy, 2009, Park et al, 2017, Zeng et al, 2017. Finally, the probabilistic formulation is very versatile, and turns several complex settings into natural extensions of the simple Gaussian ones mentioned above.…”