MIP-BOOST: Efficient and Effective $L_0$ Feature Selection for Linear Regression

Kenney, Ana; Chiaromonte, Francesca; Felici, G.

doi:10.48550/arxiv.1808.02526

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is one argument commonly used against exact (cardinality-constrained) sparse regression formulations: the sparsity parameters might not be known, in which case they need to be cross-validated hence resulting in a dramatic increase in the required computational effort. Although, in many cases, such parameters are determined by the application, Kenney et al (2018) address such concerns by proposing efficient cross validation strategies.…”

Section: Experiments On Synthetic Datasetsmentioning

confidence: 99%

Slowly Varying Regression under Sparsity

Bertsimas¹,

Digalakis²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider the problem of parameter estimation in slowly varying regression models with sparsity constraints. We formulate the problem as a mixed integer optimization problem and demonstrate that it can be reformulated exactly as a binary convex optimization problem through a novel exact relaxation. The relaxation utilizes a new equality on Moore-Penrose inverses that convexifies the non-convex objective function while coinciding with the original objective on all feasible binary points. This allows us to solve the problem significantly more efficiently and to provable optimality using a cutting plane-type algorithm. We develop a highly optimized implementation of such algorithm, which substantially improves upon the asymptotic computational complexity of a straightforward implementation. We further develop a heuristic method that is guaranteed to produce a feasible solution and, as we empirically illustrate, generates high quality warm-start solutions for the binary optimization problem. We show, on both synthetic and real-world datasets, that the resulting algorithm outperforms competing formulations in comparable times across a variety of metrics including out-of-sample predictive performance, support recovery accuracy, and false positive rate. The algorithm enables us to train models with 10,000s of parameters, is robust to noise, and able to effectively capture the underlying slowly changing support of the data generating process.

show abstract

Section: Experiments On Synthetic Datasetsmentioning

confidence: 99%

Slowly Varying Regression under Sparsity

Bertsimas¹,

Digalakis²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Since the function to minimize does not depend on k, any piece-wise linear lower approximation of c(s) computed to solve (3) for some value of k can be reused to solve the problem at another sparsity level. In recent work, Kenney et al [32] proposed a combination of implementation recipes to optimize such search procedures. As for γ, we apply the procedure described in Chu et al [11], starting with a low value γ 0 (typically scaling as 1/max i x i…”

Section: Implementation and Publicly Available Codementioning

confidence: 99%

Sparse Regression: Scalable algorithms and empirical performance

Bertsimas,

Pauphilet,

Van Parys

2019

Preprint

View full text Add to dashboard Cite

In this paper, we review state-of-the-art methods for feature selection in statistics with an applicationoriented eye. Indeed, sparsity is a valuable property and the profusion of research on the topic might have provided little guidance to practitioners. We demonstrate empirically how noise and correlation impact both the accuracy -the number of correct features selected -and the false detection -the number of incorrect features selected -for five methods: the cardinality-constrained formulation, its Boolean relaxation, 1 regularization and two methods with non-convex penalties. A cogent feature selection method is expected to exhibit a two-fold convergence, namely the accuracy and false detection rate should converge to 1 and 0 respectively, as the sample size increases. As a result, proper method should recover all and nothing but true features. Empirically, the integer optimization formulation and its Boolean relaxation are the closest to exhibit this two properties consistently in various regimes of noise and correlation. In addition, apart from the discrete optimization approach which requires a substantial, yet often affordable, computational time, all methods terminate in times comparable with the glmnet package for Lasso. We released code for methods that were not publicly implemented. Jointly considered, accuracy, false detection and computational time provide a comprehensive assessment of each feature selection method and shed light on alternatives to the Lasso-regularization which are not as popular in practice yet.

show abstract

“…When used in conjunction with warm starts from a projected gradient descent method, showed that their mixed-integer optimization approach for best subsets can be applied to problems with dimensions as large as p ≈ 1000. This development represents the first time that the best subsets estimator has been tractable for contemporary high-dimensional data after at least 50 years of literature and has paved the way for exciting new research Bertsimas et al 2019;Hastie et al 2019;Hazimeh and Mazumder 2019;Kenney et al 2019;Kreber 2019;Bertsimas and Van Parys 2020;Takano and Miyashiro 2020).…”

Section: Introductionmentioning

confidence: 99%

Robust subset selection

Thompson

2020

Preprint

View full text Add to dashboard Cite

The best subset selection (or "best subsets") estimator is a classic tool for sparse regression, and developments in mathematical optimization over the past decade have made it more computationally tractable than ever. Notwithstanding its desirable statistical properties, the best subsets estimator is susceptible to outliers and can break down in the presence of a single contaminated data point. To address this issue, we propose a robust adaption of best subsets that is highly resistant to contamination in both the response and the predictors. Our estimator generalizes the notion of subset selection to both predictors and observations, thereby achieving robustness in addition to sparsity. This procedure, which we call "robust subset selection" (or "robust subsets"), is defined by a combinatorial optimization problem for which we apply modern discrete optimization methods. We formally establish the robustness of our estimator in terms of the finite-sample breakdown point of its objective value. In support of this result, we report experiments on both synthetic and real data that demonstrate the superiority of robust subsets over best subsets in the presence of contamination. Importantly, robust subsets fares competitively across several metrics compared with popular robust adaptions of the Lasso.

show abstract

MIP-BOOST: Efficient and Effective $L_0$ Feature Selection for Linear Regression

Cited by 3 publications

References 0 publications

Slowly Varying Regression under Sparsity

Slowly Varying Regression under Sparsity

Sparse Regression: Scalable algorithms and empirical performance

Robust subset selection

Contact Info

Product

Resources

About