Sparse multinomial logistic regression: fast algorithms and generalization bounds

Abstract

Abstract-Recently developed methods for learning sparse classifiers are among the state-of-the-art in supervised learning. These methods learn classifiers that incorporate weighted sums of basis functions with sparsity-promoting priors encouraging the weight estimates to be either significantly large or exactly zero. From a learning-theoretic perspective, these methods control the capacity of the learned classifier by minimizing the number of basis functions used, resulting in better generalization. This paper presents three contributions related to learning sparse classifiers. First, we introduce a true multiclass formulation based on multinomial logistic regression. Second, by combining a bound optimization approach with a component-wise update procedure, we derive fast exact algorithms for learning sparse multiclass classifiers that scale favorably in both the number of training samples and the feature dimensionality, making them applicable even to large data sets in high-dimensional feature spaces. To the best of our knowledge, these are the first algorithms to perform exact multinomial logistic regression with a sparsity-promoting prior. Third, we show how nontrivial generalization bounds can be derived for our classifier in the binary case. Experimental results on standard benchmark data sets attest to the accuracy, sparsity, and efficiency of the proposed methods.

Automatic covariate selection in logistic models for chest pain diagnosis: A new approach

“…In [29] the MM method was applied to the least absolute deviation regression problem, leading to a viable surrogate for the absolute value penalty. By combining these ideas [8] provides an iterative algorithm for the maximization of the log-likelihood function with sparsity inducing penalty in the multinomial case -this is the method used here in its binomial form. The resulting algorithm requires no more computational resource than the iteratively re-weighted least-squares method conventionally used for solving the logistic regression problem.…”

“…The resulting algorithm requires no more computational resource than the iteratively re-weighted least-squares method conventionally used for solving the logistic regression problem. The authors of [8] have provided a fairly comprehensive, downloadable package [30], however, it is easily programmed in the Matlab TM environment [31], which is the approach adopted here.…”

“…By exploiting results from optimization theory known as optimization transfer or minorization-majorization (MM) algorithms [6,7], it has recently been possible to extend sparsity-inducing regularization to the problem of multinomial logistic regression [8] resulting in an algorithm that is no more computationally demanding than conventional iterative schemes. This offers a principled alternative to the widely used, sub-optimal step-wise regression procedures provided in a number of packages.…”

Robust varying coefficient model using L1 regularization

“…with equality if and only if β = β * (Krishnapuram et al, 2005) and L(β|β * , x) is differentiable with respect to β. At th iteration of IRWLS procedure, we have…”

Semi-supervised hyperspectral image classification based on a Markov random field and sparse multinomial logistic regression

“…Sparse multinomial logistic regression methods are available [11]. More recently, the introduction of the LORSAL (logistic regression via variable splitting and augmented Lagrangian) algorithm [12] has open the door to deal with larger data sets and number of classes.…”