Learning to Optimize Computational Resources: Frugal Training with Generalization Guarantees

Balcan, Maria-Florina; Sandholm, Tüomas; Vitercik, Ellen

doi:10.1609/aaai.v34i04.5721

Cited by 11 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We plot a similar curve for the test performance of the oracle algorithm selector which always selects the optimal parameter setting from the portfolio. Specifically, for each portfolio size κ ∈ [10], let f * κ be the oracle algorithm selector f * κ (z) = argmin ρ 1 ,...,ρκ u ρ i (z). Given a test set S t ∼ D Nt , we define the average test performance of f * κ as…”

Section: Methodsmentioning

confidence: 99%

Generalization in portfolio-based algorithm selection

Balcan¹,

Sandholm²,

Vitercik³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Portfolio-based algorithm selection has seen tremendous practical success over the past two decades. This algorithm configuration procedure works by first selecting a portfolio of diverse algorithm parameter settings, and then, on a given problem instance, using an algorithm selector to choose a parameter setting from the portfolio with strong predicted performance. Oftentimes, both the portfolio and the algorithm selector are chosen using a training set of typical problem instances from the application domain at hand. In this paper, we provide the first provable guarantees for portfolio-based algorithm selection. We analyze how large the training set should be to ensure that the resulting algorithm selector's average performance over the training set is close to its future (expected) performance. This involves analyzing three key reasons why these two quantities may diverge: 1) the learning-theoretic complexity of the algorithm selector, 2) the size of the portfolio, and 3) the learning-theoretic complexity of the algorithm's performance as a function of its parameters. We introduce an end-to-end learning-theoretic analysis of the portfolio construction and algorithm selection together. We prove that if the portfolio is large, overfitting is inevitable, even with an extremely simple algorithm selector. With experiments, we illustrate a tradeoff exposed by our theoretical analysis: as we increase the portfolio size, we can hope to include a well-suited parameter setting for every possible problem instance, but it becomes impossible to avoid overfitting.

show abstract

Section: Methodsmentioning

confidence: 99%

Generalization in portfolio-based algorithm selection

Balcan¹,

Sandholm²,

Vitercik³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our guarantees are configuration-procedure-agnostic: no matter how one tunes the parameters using the training set, we bound the difference between the resulting parameter setting's performance on average over the training set and its expected performance on unseen instances. A related line of research has provided learning-based algorithm configuration procedures with provable guarantees [12,27,28,44,45]. Unlike the results in this paper, their guarantees are not configuration-procedure-agnostic: they apply to the specific configuration procedures they propose.…”

Section: Introductionmentioning

confidence: 92%

Refined bounds for algorithm configuration: The knife-edge of dual class approximability

Balcan,

Sandholm,

Vitercik

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Automating algorithm configuration is growing increasingly necessary as algorithms come with more and more tunable parameters. It is common to tune parameters using machine learning, optimizing performance metrics such as runtime and solution quality. The training set consists of problem instances from the specific domain at hand. We investigate a fundamental question about these techniques: how large should the training set be to ensure that a parameter's average empirical performance over the training set is close to its expected, future performance? We answer this question for algorithm configuration problems that exhibit a widely-applicable structure: the algorithm's performance as a function of its parameters can be approximated by a "simple" function. We show that if this approximation holds under the L ∞norm, we can provide strong sample complexity bounds. On the flip side, if the approximation holds only under the L p -norm for p < ∞, it is not possible to provide meaningful sample complexity bounds in the worst case. We empirically evaluate our bounds in the context of integer programming, one of the most powerful tools in computer science. Via experiments, we obtain sample complexity bounds that are up to 700 times smaller than the previously best-known bounds [7].

show abstract

“…However, these methods are typically limited to specific CO problems in which a heuristic solution can be easily constructed, and scaling to large-size instances is an issue. On the other hand, since a wide range of constrained CO problems can be formulated into a MIP model, there has also been increasing interest in learning decision rules to improve MIP algorithms [7,[18][19][20][21][22]. While it is shown that this direction has a great potential to improve the state-of-the-art of MIP algorithms, convincing generalization performances, and transfer learning across instances have not been fully tackled yet.…”

Section: Representation Learning For Comentioning

confidence: 99%

A machine learning framework for neighbor generation in metaheuristic search

Liu

Perreault

Hertz

et al. 2023

Front. Appl. Math. Stat.

View full text Add to dashboard Cite

This paper presents a methodology for integrating machine learning techniques into metaheuristics for solving combinatorial optimization problems. Namely, we propose a general machine learning framework for neighbor generation in metaheuristic search. We first define an efficient neighborhood structure constructed by applying a transformation to a selected subset of variables from the current solution. Then, the key of the proposed methodology is to generate promising neighbors by selecting a proper subset of variables that contains a descent of the objective in the solution space. To learn a good variable selection strategy, we formulate the problem as a classification task that exploits structural information from the characteristics of the problem and from high-quality solutions. We validate our methodology on two metaheuristic applications: a Tabu Search scheme for solving a Wireless Network Optimization problem and a Large Neighborhood Search heuristic for solving Mixed-Integer Programs. The experimental results show that our approach is able to achieve a satisfactory trade-offs between the exploration of a larger solution space and the exploitation of high-quality solution regions on both applications.

show abstract

Learning to Optimize Computational Resources: Frugal Training with Generalization Guarantees

Cited by 11 publications

References 11 publications

Generalization in portfolio-based algorithm selection

Generalization in portfolio-based algorithm selection

Refined bounds for algorithm configuration: The knife-edge of dual class approximability

A machine learning framework for neighbor generation in metaheuristic search

Contact Info

Product

Resources

About