Generalization in portfolio-based algorithm selection

Abstract

Portfolio-based algorithm selection has seen tremendous practical success over the past two decades. This algorithm configuration procedure works by first selecting a portfolio of diverse algorithm parameter settings, and then, on a given problem instance, using an algorithm selector to choose a parameter setting from the portfolio with strong predicted performance. Oftentimes, both the portfolio and the algorithm selector are chosen using a training set of typical problem instances from the application domain at hand. In this paper, we provide the first provable guarantees for portfolio-based algorithm selection. We analyze how large the training set should be to ensure that the resulting algorithm selector's average performance over the training set is close to its future (expected) performance. This involves analyzing three key reasons why these two quantities may diverge: 1) the learning-theoretic complexity of the algorithm selector, 2) the size of the portfolio, and 3) the learning-theoretic complexity of the algorithm's performance as a function of its parameters. We introduce an end-to-end learning-theoretic analysis of the portfolio construction and algorithm selection together. We prove that if the portfolio is large, overfitting is inevitable, even with an extremely simple algorithm selector. With experiments, we illustrate a tradeoff exposed by our theoretical analysis: as we increase the portfolio size, we can hope to include a well-suited parameter setting for every possible problem instance, but it becomes impossible to avoid overfitting.

Generalization in Portfolio-Based Algorithm Selection

“…For example, when algorithmic performance as a function of the parameters is piecewise constant, there are only a finite number of meaningfully different parameter values to choose among-one per piece. Then, since z∈S max ρ∈ P u ρ (z) is a submodular function of the portfolio P, we can use a greedy algorithm to select P, and we obtain α = 1 − 1 e and β = 0, as we prove in the full version (Balcan, Sandholm, and Vitercik 2020). Alternatively, integer programming could be used to select the optimal portfolio from the finite set of candidate parameter values, in which case we would obtain α = 1 and β = 0.…”

“…We provide a general bound on Pdim (U F ), which allows us to bound Equation (1). The proof is in the full version (Balcan, Sandholm, and Vitercik 2020). Theorem 3.4.…”

“…, T κ ). The full proof of the following lemma is in the full version (Balcan, Sandholm, and Vitercik 2020). Lemma 4.2.…”

“…where each x i represents a cluster center. We define the selector f ρ,X (z) = ρ i with i = argmin j∈[κ] x j − φ(z) p for some p -norm with p ≥ 1, and the class (Balcan, Sandholm, and Vitercik 2020). Lemma 4.4.…”

“…The following theorem bounds the difference between the expected performance of the selector f and an oracle that selects an optimal selector and an optimal portfolio. The proof is in the full version (Balcan, Sandholm, and Vitercik 2020).…”