Sparse kernel logistic regression based on L 1/2 regularization

Abstract

The sparsity driven classification technologies have attracted much attention in recent years, due to their capability of providing more compressive representations and clear interpretation. Two most popular classification approaches are support vector machines (SVMs) and kernel logistic regression (KLR), each having its own advantages. The sparsification of SVM has been well studied, and many sparse versions of 2-norm SVM, such as 1-norm SVM (1-SVM), have been developed. But, the sparsification of KLR has been less studied. The existing sparsification of KLR is mainly based on L 1 norm and L 2 norm penalties, which leads to the sparse versions that yield solutions not so sparse as it should be. A very recent study on L 1/2 regularization theory in compressive sensing shows that L 1/2 sparse modeling can yield solutions more sparse than those of 1 norm and 2 norm, and, furthermore, the model can be efficiently solved by a simple iterative thresholding procedure. The objective function dealt with in L 1/2 regularization theory is, however, of square form, the gradient of which is linear in its variables (such an objective function is the so-called linear gradient function). In this paper, through extending the linear gradient function of L 1/2 regularization framework to the logistic function, we propose a novel sparse version of KLR, the 1/2 quasi-norm kernel logistic regression (1/2-KLR). The version integrates advantages of KLR and L 1/2 regularization, and defines an efficient implementation scheme of sparse KLR. We suggest a fast iterative thresholding algorithm for 1/2-KLR and prove its convergence. We provide a series of simulations to demonstrate that 1/2-KLR can often obtain more sparse solutions than the existing sparsity driven versions of KLR, at the same or better accuracy level. The conclusion is also true even in comparison with sparse SVMs (1-SVM and 2-SVM). We show an exclusive advantage of 1/2-KLR that the regularization parameter in the algorithm can be adaptively set whenever the sparsity (correspondingly, the number of support vectors) is given, which suggests a methodology of comparing sparsity promotion capability of different sparsity driven classifiers. As an illustration of benefits of 1/2-KLR, we give two applications of 1/2-KLR in semi-supervised learning, showing that 1/2-KLR can be successfully applied to the classification tasks in which only a few data are labeled.

<formula formulatype="inline"><tex Notation="TeX">$L_{1/2}$</tex> </formula> Regularization: Convergence of Iterative Half Thresholding Algorithm

“…is monotonically decreasing with respect to , there exists a such that and . We will proceed to show that whenever , it holds Actually, we have (27) By Proposition 2, for any , (28) and for any ,…”

“…Furthermore, through developing a thresholding representation theory, an iterative half thresholding algorithm (called half algorithm in brief) was proposed in [20] for fast solution to regularization. Inspired by the well developed theoretical properties and the fast algorithm, regularization has been successfully used to many applications including hyperspectral unmixing [24], synthetic aperture radar (SAR) imaging [25], [26], machine learning [27], [28], gene selection [29] and practical engineering [30].…”

Multinomial Regression with Elastic Net Penalty and Its Grouping Effect in Gene Selection

“…By combing the multiclass elastic net penalty (18) with the multinomial likelihood loss function (17), we propose the following multinomial regression model with the elastic net penalty:…”

“…Note that the logistic loss function not only has good statistical significance but also is second order differentiable. Hence, the regularized logistic regression optimization models have been successfully applied to binary classification problem [15][16][17][18][19]. Multinomial regression can be obtained when applying the logistic regression to the multiclass classification problem.…”

Kernel selection with spectral perturbation stability of kernel matrix

“…Kernel methods [1,2] have been successfully used in pattern recognition and machine learning. Since the performance of kernel methods greatly depends on the selection of the kernel function, the kernel selection problem becomes an important topic in kernel methods [3][4][5].…”