A Road to Classification in High Dimensional Space: The Regularized Optimal Affine Discriminant

Abstract

Summary. For high dimensional classification, it is well known that naively performing the Fisher discriminant rule leads to poor results due to diverging spectra and accumulation of noise. Therefore, researchers proposed independence rules to circumvent the diverging spectra, and sparse independence rules to mitigate the issue of accumulation of noise. However, in biological applications, often a group of correlated genes are responsible for clinical outcomes, and the use of the covariance information can significantly reduce misclassification rates. In theory the extent of such error rate reductions is unveiled by comparing the misclassification rates of the Fisher discriminant rule and the independence rule. To materialize the gain on the basis of finite samples, a regularized optimal affine discriminant (ROAD) is proposed. The ROAD selects an increasing number of features as the regularization relaxes. Further benefits can be achieved when a screening method is employed to narrow the feature pool before applying the ROAD method. An efficient constrained co‐ordinate descent algorithm is also developed to solve the associated optimization problems. Sampling properties of oracle type are established. Simulation studies and real data analysis support our theoretical results and demonstrate the advantages of the new classification procedure under a variety of correlation structures. A delicate result on continuous piecewise linear solution paths for the ROAD optimization problem at the population level justifies the linear interpolation of the constrained co‐ordinate descent algorithm.

A New Reduced-Rank Linear Discriminant Analysis Method and Its Applications

“…Targeting on cancer classification and other modern applications, a great number of high dimensional classification techniques have been invented and studied recently; see Hastie et al (2009) for an extensive introduction, and Witten & Tibshirani (2011) ;Cai & Liu (2011);Fan et al (2012); Mai et al (2012) for some recent developments. Although these contemporary classification tools can be applied to the high dimensional data, most of them rely on some strong assumptions.…”

Recipes for sparse LDA of horizontal data

“…Fan et al [13] argued that ignoring the covariances (the off-diagonal entries in W) as suggested by Bickel and Levina [2] may not be a good idea. In order to avoid redefining Fisher's LDA for singular W, Fan et al [13] proposed working with (minimizing) the classification error instead of the Fisher's LDA ratio (1).…”

“…In order to avoid redefining Fisher's LDA for singular W, Fan et al [13] proposed working with (minimizing) the classification error instead of the Fisher's LDA ratio (1). The method is called for short ROAD (from Regularized Optimal Affine Discriminant) and is developed for two groups.…”

“…Thus, the ROAD minimizer a is sought subject to a LASSO-type constraint introduced as a penalty term, i.e. : min Further on, Fan et al [13] replace the affine constrained problem (12) by a quadratic penalty term, which results in the following unconstraint problem:…”

Penalized Regression Versus Random Forest Model in Analyzing High Dimensional Proteomic Data: Diagnosis of IgA Nephropathy

“…In high dimensional data sets, the number of variables is very large whereas, and due to excessive costs, the sample size is typically small (1). Several studies have referred to challenges occurring in high dimensional settings as "curse of dimensionality" (2).…”