ASlib: A benchmark library for algorithm selection

Abstract

The task of algorithm selection involves choosing an algorithm from a set of algorithms on a per-instance basis in order to exploit the varying performance of algorithms over a set of instances. The algorithm selection problem is attracting increasing attention from researchers and practitioners in AI. Years of fruitful applications in a number of domains have resulted in a large amount of data, but the community lacks a standard format or repository for this data. This situation makes it difficult to share and compare different approaches effectively, as is done in other, more established fields. It also unnecessarily hinders new researchers who want to work in this area. To address this problem, we introduce a standardized format for representing algorithm selection scenarios and a repository that contains a growing number of data sets from the literature. Our format has been designed to be able to express a wide variety of different scenarios. To demonstrate the breadth and power of our platform, we describe a study that builds and evaluates algorithm selection models through a common interface. The results display the potential of algorithm selection to achieve significant performance improvements across a broad range of problems and algorithms.• A human-readable README file explains the origin and meaning of the scenario, as well as the process of data generation.Optional Data.• The feature costs file contains the costs of the feature groups, i.e., sets of features computed together.• The ground truth file specifies information on the instances and their respective solutions (e.g., SAT or UNSAT).• The literature references file in BibTeX format includes information on the context in which the data set was generated and previous studies in which it was used. Algorithm Selection Scenarios Provided in ASlib Release 2.0The set of algorithm selection scenarios in release version 2.0 of our library, shown in Table 2, has been assembled to represent a diverse set of selection problem settings that covers a wide range of problem domains, types of algorithms, 7 http://www.satcompetition.org/ 8 http://www.qbflib.org/index_eval.php 9 http://qbf.satisfiability.org/gallery/

Meta-QSAR: a large-scale application of meta-learning to drug design and discovery

“…Finally, we would also like to explore other algorithm selection techniques beyond Random Forests. To this end, we plan to export our experiments from OpenML to an ASlib scenario (Bischl et al 2016), where many algorithm selection techniques could be compared.…”

An Extensive Evaluation of Portfolio Approaches for Constraint Satisfaction Problems

“…Clearly, comparing them all is a daunting task. To address this problem, the Algorithm Selection Library (ASlib) [9] has been recently introduced. ASlib provides a standardized format for representing very heterogeneous portfolio scenarios with the aim of effectively sharing and comparing different approaches.…”

“…However, for making our experiments reproducible and comparable, we submitted the evaluation scenario to the Algorithm Selection library [9]. Indeed, in addition to the approaches evaluated in this paper, a plethora of other CSP portfolio approaches have been proposed in the literature [32,12,46,18,2].…”

“…We want instead to shed further light on CSP portfolio approaches by means of empirical evaluations. In this regard, we submitted the data and the results we computed to the Algorithm Selection library [9], an open-access library providing a standardized format for representing, evaluating, and comparing different portfolio approaches without the effort of rebuilding all the experimental environment.…”

The ICON Challenge on Algorithm Selection

“…The challenge leveraged the ASlib benchmark library for algorithm selection (Bischl et al 2016). We used thirteen scenarios, drawn from prominent publications, in release 1.0.…”