Trading Accuracy for Sparsity in Optimization Problems with Sparsity Constraints

Abstract

Abstract. We study the problem of minimizing the expected loss of a linear predictor while constraining its sparsity, i.e., bounding the number of features used by the predictor. While the resulting optimization problem is generally NP-hard, several approximation algorithms are considered. We analyze the performance of these algorithms, focusing on the characterization of the trade-off between accuracy and sparsity of the learned predictor in different scenarios. Key words. sparsity, linear prediction AMS subject classifications. 68T99, 68W40DOI. 10.1137/090759574 1. Introduction. In statistical and machine learning applications, although many features might be available for use in a prediction task, it is often beneficial to use only a small subset of the available features. Predictors that use only a small subset of features require a smaller memory footprint and can be applied faster. Furthermore, in applications such as medical diagnostics, obtaining each possible "feature" (e.g., test result) can be costly, and so a predictor that uses only a small number of features is desirable, even at the cost of a small degradation in performance relative to a predictor that uses more features.These applications lead to optimization problems with sparsity constraints. Focusing on linear prediction, it is generally NP-hard to find the best predictor subject to a sparsity constraint, i.e., a bound on the number of features used [7,19]. In this paper we show that by compromising on prediction accuracy, one can compute sparse predictors efficiently. Our main goal is to understand the precise trade-off between accuracy and sparsity and how this trade-off depends on properties of the underlying optimization problem.We now formally define our problem setting. A linear predictor is a mapping x → φ( w, x ), where x ∈ X

Greedy Approximation in Convex Optimization

“…It is pointed out in [46] that in many engineering applications, researchers are interested in an approximate solution of problem (2) as a linear combination of a few elements from a given system D of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms (see, for instance, [1,[3][4][5]11,12,17,19,[22][23][24]34,45,46]). We refer the reader to the papers [1] and [23] for concise surveys of recent results on greedy algorithms from the point of view of convex optimization and signal processing.…”

“…It is used for exact recovery of sparse signals and for approximation of signals by sparse ones. An analog of the WCGA in convex optimization was introduced in [34] under the name Fully Corrective Forward Greedy Selection. The WRGA is the approximation theory analog of the classical Frank-Wolfe algorithm, introduced in [21] and studied in many papers (see, for instance, [12,14,17,18,22,23]).…”

“…6). Better rate of convergence results can be obtained under stronger assumptions on E for dictionaries satisfying certain conditions [32,34,44].…”

On Sparse Feature Attacks in Adversarial Learning

“…Existing literatures [15], [16] indicates that in a classical classification environment, when 1 regularizer is applied, there will always be a trade-off between the enforced sparsity and the accuracy obtained. However, depending on the dataset itself, the overall performance of an 1 regularized classifier can sometimes beat an 2 regularized classifier [17] in terms of both bias and variance.…”

Feature engineering and tree modeling for author-paper identification challenge

“…A popular method to solve this problem is through decision tree learning (such as CART [6] and C4.5 [8]), which has an important advantage for handling heterogenous data with case when different features come from different sources.…”