MachSMT: A Machine Learning-based Algorithm Selector for SMT Solvers

Scott, Joseph; Niemetz, Aina; Preiner, Mathias; Nejati, Saeed

doi:10.1007/978-3-030-72013-1_16

Cited by 15 publications

(5 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As mentioned previously, it is well established that the state of the art for solving challenging computational problems (e.g., SAT [58], planning [54], minimum vertex cover [59], answer set programming [48], satisfiability modulo theories [60], etc.) is not defined by a single solver, but rather by a collection of non-dominated solvers with complementary strengths.…”

Section: Sparkle As a Competition Platformmentioning

confidence: 97%

Sparkle: Toward Accessible Meta-Algorithmics for Improving the State of the Art in Solving Challenging Problems

Blom

Hoos

Luo

et al. 2022

IEEE Trans. Evol. Computat.

View full text Add to dashboard Cite

Many fields of computational science advance through improvements in the algorithms used for solving key problems. These advancements are often facilitated by benchmarks and competitions that enable performance comparisons and rankings of solvers. Simultaneously, meta-algorithmic techniques, such as automated algorithm selection and configuration, enable performance improvements by utilising the complementary strengths of different algorithms or configurable algorithm components. In fact, meta-algorithms have become major drivers in advancing the state of the art in solving many prominent computational problems.However, meta-algorithmic techniques are complex and difficult to use correctly, while their incorrect use may reduce their efficiency, or in extreme cases, even lead to performance losses. Here, we introduce the Sparkle platform, which aims to make meta-algorithmic techniques more accessible to non-expert users, and to make these techniques more broadly available in the context of competitions, to further enable the assessment and advancement of the true state of the art in solving challenging computational problems. To achieve this, Sparkle implements standard protocols for algorithm selection and configuration that support easy and correct use of these techniques. Following an experiment, Sparkle generates a report containing results, problem instances, algorithms and other relevant information, for convenient use in scientific publications.

show abstract

Section: Sparkle As a Competition Platformmentioning

confidence: 97%

Sparkle: Toward Accessible Meta-Algorithmics for Improving the State of the Art in Solving Challenging Problems

Blom

Hoos

Luo

et al. 2022

IEEE Trans. Evol. Computat.

View full text Add to dashboard Cite

show abstract

“…For example, in AI planning, Planzilla (Rizzini, Fawcett, Vallati, Gerevini, & Hoos, 2017) and its improved variants (model-based approaches) were all inspired by the random forests and regression techniques proposed by SATZilla/Zilla. Similarly, for Satisfiability Modulo Theories (SMT) problems, MachSMT (Scott, Niemetz, Preiner, Nejati, & Ganesh, 2021) was recently introduced and its essential parts also rely on random forests. The main difference between the model-based Planzilla selector and MachSMT is that the first one chooses solvers minimizing the ratio between solved instances and solving time, while the latter only considers the solving time of candidate solvers.…”

Section: Related Workmentioning

confidence: 99%

sunny-as2: Enhancing SUNNY for Algorithm Selection

Liu¹,

Amadini

Gabbrielli

et al. 2021

jair

View full text Add to dashboard Cite

SUNNY is an Algorithm Selection (AS) technique originally tailored for Constraint Programming (CP). SUNNY is based on the k-nearest neighbors algorithm and enables one to schedule, from a portfolio of solvers, a subset of solvers to be run on a given CP problem. This approach has proved to be effective for CP problems. In 2015, the ASlib benchmarks were released for comparing AS systems coming from disparate fields (e.g., ASP, QBF, and SAT) and SUNNY was extended to deal with generic AS problems. This led to the development of sunny-as, a prototypical algorithm selector based on SUNNY for ASlib scenarios. A major improvement of sunny-as, called sunny-as2, was then submitted to the Open Algorithm Selection Challenge (OASC) in 2017, where it turned out to be the best approach for the runtime minimization of decision problems. In this work we present the technical advancements of sunny-as2, by detailing through several empirical evaluations and by providing new insights. Its current version, built on the top of the preliminary version submitted to OASC, is able to outperform sunny-as and other state-of-the-art AS methods, including those who did not attend the challenge.

show abstract

“…Although one algorithm we apply is k-NN, we use it to select sequences of solvers and apply it in combination with the time-prediction algorithm. MachSMT [35] is a pre-trained tool like SatZilla but for SMT. Like SatZilla, MachSMT pre-trains to learn an empirical hardness model, then used to predict solving time for a given query.…”

Section: Related Workmentioning

confidence: 99%

“…MachSMT [35] uses a neural network to select which solver to run on a given query. Table 2 shows the performance of MachSMT and MedleySolver on the same benchmarks as in Table 1 but with 2/5 of the queries set aside for MachSMT to train on per set.…”

Section: Rq2: Comparison With State-of-the-artmentioning

confidence: 99%

MedleySolver: Online SMT Algorithm Selection

Pimpalkhare

Mora

Polgreen

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Satisfiability modulo theories (SMT) solvers implement a wide range of optimizations that are often tailored to a particular class of problems, and that differ significantly between solvers. As a result, one solver may solve a query quickly while another might be flummoxed completely. Predicting the performance of a given solver is difficult for users of SMT-driven applications, particularly when the problems they have to solve do not fall neatly into a well-understood category. In this paper, we propose an online algorithm selection framework for SMT called MedleySolver that predicts the relative performances of a set of SMT solvers on a given query, distributes time amongst the solvers, and deploys the solvers in sequence until a solution is obtained. We evaluate MedleySolver against the best available alternative, an offline learning technique, in terms of pure performance and practical usability for a typical SMT user. We find that with no prior training, MedleySolver solves 93.9% of the queries solved by the virtual best solver selector achieving 59.8% of the par-2 score of the most successful individual solver, which solves 87.3%. For comparison, the best available alternative takes longer to train than MedleySolver takes to solve our entire set of 2000 queries.

show abstract

MachSMT: A Machine Learning-based Algorithm Selector for SMT Solvers

Cited by 15 publications

References 54 publications

Sparkle: Toward Accessible Meta-Algorithmics for Improving the State of the Art in Solving Challenging Problems

Sparkle: Toward Accessible Meta-Algorithmics for Improving the State of the Art in Solving Challenging Problems

sunny-as2: Enhancing SUNNY for Algorithm Selection

MedleySolver: Online SMT Algorithm Selection

Contact Info

Product

Resources

About