Spectral Regression: A Unified Approach for Sparse Subspace Learning

Cai, Deng; He, Xiaofei; Han, Jiawei

doi:10.1109/icdm.2007.89

Cited by 184 publications

(119 citation statements)

References 17 publications

Supporting

Mentioning

119

Contrasting

Order By: Relevance

“…For a multi-class problem, the authors suggested the one against the rest scheme by considering all two-class problems. Recent studies [4][5][6][7]9] show that various linear dimensionality reduction algorithms can be formulated as regression problems and thus have efficient computational solutions. Particularly, our previous work [6] has demonstrated that LDA can be formulated as a regression problem and be efficiently solved.…”

Section: Introductionmentioning

confidence: 99%

Speed up kernel discriminant analysis

2010

Self Cite

View full text Add to dashboard Cite

Linear Discriminant Analysis (LDA) has been a popular method for dimensionality reduction which preserves class separability. The projection vectors are commonly obtained by maximizing the between class covariance and simultaneously minimizing the within class covariance. LDA can be performed either in the original input space or in the reproducing kernel Hilbert space (RKHS) into which data points are mapped, which leads to Kernel Discriminant Analysis (KDA). When the data are highly nonlinear distributed, KDA can achieve better performance than LDA. However, computing the projective functions in KDA involves eigendecomposition of kernel matrix, which is very expensive when a large number of training samples exist. In this paper, we present a new algorithm for kernel discriminant analysis, called Spectral Regression Kernel Discriminant Analysis (SRKDA). By using spectral graph analysis, SRKDA casts discriminant analysis into a regression framework which facilitates both efficient computation and the use of regularization techniques. Specifically, SRKDA only needs to solve a set of regularized regression problems and there is no eigenvector computation involved, which is a huge save of computational cost. The new formulation makes it very easy to develop incremental version of the algorithm which can

show abstract

Section: Introductionmentioning

confidence: 99%

Speed up kernel discriminant analysis

2010

Self Cite

View full text Add to dashboard Cite

show abstract

“…PCA, LDA and Locality Preserving Projection (LPP) [12], sparse LDA (SLDA) [5]. We also compare it with the feature selection methods, e.g., Fisher score (FS) and Linear Discriminant Feature Selection (LDFS) [16].…”

Section: Methodsmentioning

confidence: 99%

“…LDA has been successfully applied to face recognition [2]. Following LDA, many incremental works have been done, e.g., Uncorrelated LDA and Orthogonal LDA [26], Local LDA [24], Semi-supervised LDA [4] and Sparse LDA [17] [5]. Note that all these methods suffer from the weakness of using all the original features to learn the subspace.…”

Section: Linear Discriminant Analysismentioning

confidence: 99%

“…In order to pursue sparsity and interpretability in LDA, [17] proposed both exact and greedy algorithms for binary class sparse LDA as well as its spectral bound. For multi-class problem, [5] proposed a sparse LDA (SLDA) based on ℓ 1 -norm regularized Spectral Regression,…”

Section: Related Workmentioning

confidence: 99%

“…And LDA has received great success in face recognition [2], which is known as Fisher Face. In the past decades, both Fisher score and LDA have been studied extensively [10] [20] [26] [24] [4] [17] [5]. However, they study Fisher score or LDA independently, ignoring the close relation between them.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Linear Discriminant Dimensionality Reduction

Han

2011

Machine Learning and Knowledge Discovery in Databases

Self Cite

View full text Add to dashboard Cite

Abstract. Fisher criterion has achieved great success in dimensionality reduction. Two representative methods based on Fisher criterion are Fisher Score and Linear Discriminant Analysis (LDA). The former is developed for feature selection while the latter is designed for subspace learning. In the past decade, these two approaches are often studied independently. In this paper, based on the observation that Fisher score and LDA are complementary, we propose to integrate Fisher score and LDA in a unified framework, namely Linear Discriminant Dimensionality Reduction (LDDR). We aim at finding a subset of features, based on which the learnt linear transformation via LDA maximizes the Fisher criterion. LDDR inherits the advantages of Fisher score and LDA and is able to do feature selection and subspace learning simultaneously. Both Fisher score and LDA can be seen as the special cases of the proposed method. The resultant optimization problem is a mixed integer programming, which is difficult to solve. It is relaxed into a L2,1-norm constrained least square problem and solved by accelerated proximal gradient descent algorithm. Experiments on benchmark face recognition data sets illustrate that the proposed method outperforms the state of the art methods arguably.

show abstract