Best subset selection via a modern optimization lens

Abstract

In the last twenty-five years , algorithmic advances in integer optimization combined with hardware improvements have resulted in an astonishing 200 billion factor speedup in solving Mixed Integer Optimization (MIO) problems. We present a MIO approach for solving the classical best subset selection problem of choosing k out of p features in linear regression given n observations. We develop a discrete extension of modern first order continuous optimization methods to find high quality feasible solutions that we use as warm starts to a MIO solver that finds provably optimal solutions. The resulting algorithm (a) provides a solution with a guarantee on its suboptimality even if we terminate the algorithm early, (b) can accommodate side constraints on the coefficients of the linear regression and (c) extends to finding best subset solutions for the least absolute deviation loss function. Using a wide variety of synthetic and real datasets, we demonstrate that our approach solves problems with n in the 1000s and p in the 100s in minutes to provable optimality, and finds near optimal solutions for n in the 100s and p in the 1000s in minutes. We also establish via numerical experiments that the MIO approach performs better than Lasso and other popularly used sparse learning procedures, in terms of achieving sparse solutions with good predictive power.

Best subset binary prediction

“…Our implementation builds on the method of mixed integer optimization (in particular, Bertsimas, King, and Mazumder (2016), Florios and Skouras (2008), and Kitagawa and Tetenov (2015)) and present two alternative solution methods that complement each other. The values (M i ) n i=1 can be computed by formulating the maximization problem in (4.4) as linear programming problems, which can be easily and efficiently solved by modern numerical software such as MAT-LAB.…”

“…Using the terminology used in Bertsimas, King, and Mazumder (2016), we shall refer to these refined MIO representations as the warm-start MIO formulations of the PRESCIENCE problem. The value of τ is treated as a tuning parameter for solving the warm-start MIO problems.…”

“…They demonstrated that the MIO approach can efficiently deliver a provably optimal solution to the resulting 0 -norm constrained risk minimization problem for a variety of datasets with practical problem size. Our implementation of the best subset maximum score prediction rules combines insights from Bertsimas, King, and Mazumder (2016), Florios and Skouras (2008), and Kitagawa and Tetenov (2015). We present two alternative MIO solution methods that complement each other.…”

Best Subset Selection for Eliminating Multicollinearity

“…Penalized regression helps prevent regression models from overfitting noisy datasets and, accordingly, is effective for achieving high predictive performance. However, the penalty functions produce biased estimates, which are undesirable from the standpoint of model interpretation [5,6]. This paper focuses on subset selection, which is a simple but effective approach for eliminating multicollinearity.…”

Scalability of DL_POLY on High Performance Computing Platform

“…To provide more accurate information about the performance of HPC systems, the TOP500 list 2 was launched in 1993 to address the issues of fastest supercomputers in the world. Most recently, the list of supercomputers is published twice every year to determine the 500 most powerful computers in the world (J Dongarra, 2004;Oyanagi, 2002;Strohmaier et al, 2005;Kindratenko & Trancoso, 2011;Bertsimas, King, & Mazumder, 2016).…”