sunny-as2: Enhancing SUNNY for Algorithm Selection

Abstract


 
 
 SUNNY is an Algorithm Selection (AS) technique originally tailored for Constraint Programming (CP). SUNNY is based on the k-nearest neighbors algorithm and enables one to schedule, from a portfolio of solvers, a subset of solvers to be run on a given CP problem. This approach has proved to be effective for CP problems.
 In 2015, the ASlib benchmarks were released for comparing AS systems coming from disparate fields (e.g., ASP, QBF, and SAT) and SUNNY was extended to deal with generic AS problems. This led to the development of sunny-as, a prototypical algorithm selector based on SUNNY for ASlib scenarios. A major improvement of sunny-as, called sunny-as2, was then submitted to the Open Algorithm Selection Challenge (OASC) in 2017, where it turned out to be the best approach for the runtime minimization of decision problems.
 In this work we present the technical advancements of sunny-as2, by detailing through several empirical evaluations and by providing new insights. Its current version, built on the top of the preliminary version submitted to OASC, is able to outperform sunny-as and other state-of-the-art AS methods, including those who did not attend the challenge.
 
 

On the evaluation of (meta-)solver approaches

“…An interesting outcome reported in Liu et al (2021) was the profound difference between the closed gap and the MiniZinc challenge scores. Liu et al compared the performance of six meta-solvers approaches across 15 decision-problems scenarios taken from ASlib (Bischl et al, 2016) and coming from heterogeneous domains such as Answer-Set Programming, Constraint Programming, Quantified Boolean Formula, Boolean Satisfiability.…”

“…An initial clue of why this happens is revealed in Liu et al (2021), where a parametric version of MZNC score is used. In practice, Def.…”

“…Tab. 2 reports, using the data of (Liu et al, 2021), for each meta-solver s in a scenario (I, S, τ ) the average speedup computed as…”

“…We believe that further study on this issue is necessary because often meta-solvers are evaluated on heterogeneous scenarios, characterized by a different number of problems, different timeouts, and different individual solvers from which the meta-solvers approaches are built. In this paper, starting from some surprising results presented by Liu, Amadini, Mauro, and Gabbrielli (2021) showing dramatic ranking changes with different, but reasonable, metrics we would like to draw more attention on the evaluation of meta-solvers approaches by shedding some light on the strengths and weaknesses of different metrics.…”

On the Evaluation of (Meta-)solver Approaches

“…Starting from some surprising results presented by Liu, Amadini, Mauro, and Gabbrielli (2021) showing dramatic ranking changes with different but reasonable metrics, we would like to draw more attention to the evaluation of meta-solver approaches by shedding some light on the strengths and weaknesses of different metrics. Unsurprisingly, some of the findings we report here also apply to the evaluation of individual solvers.…”