Predicting SMT Solver Performance for Software Verification

Healy, Andrew; Monahan, Rosemary; Power, James F.

doi:10.4204/eptcs.240.2

Cited by 12 publications

(7 citation statements)

References 32 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the setting of SMT solver applications, symbolic execution tools have used algorithm selection strategies [64] and portfolio strategies [33] for the specific classes of instances within the context of the bit-vector theory. This would be an ideal use case of MachSMT, since we provide a more complete solution.…”

Section: Algorithm Selection For Logic Solvers and Their Applicationsmentioning

confidence: 99%

MachSMT: A Machine Learning-based Algorithm Selector for SMT Solvers

Scott

Niemetz

Preiner

et al. 2021

Tools and Algorithms for the Construction and Analysis of Systems

View full text Add to dashboard Cite

In this paper, we present MachSMT, an algorithm selection tool for Satisfiability Modulo Theories (SMT) solvers. MachSMT supports the entirety of the SMT-LIB language. It employs machine learning (ML) methods to construct both empirical hardness models (EHMs) and pairwise ranking comparators (PWCs) over state-of-the-art SMT solvers. Given an SMT formula $$\mathcal {I}$$ I as input, MachSMT leverages these learnt models to output a ranking of solvers based on predicted run time on the formula $$\mathcal {I}$$ I . We evaluate MachSMT on the solvers, benchmarks, and data obtained from SMT-COMP 2019 and 2020. We observe MachSMT frequently improves on competition winners, winning $$54$$ 54 divisions outright and up to a $$198.4$$ 198.4 % improvement in PAR-2 score, notably in logics that have broad applications (e.g., BV, LIA, NRA, etc.) in verification, program analysis, and software engineering. The MachSMT tool is designed to be easily tuned and extended to any suitable solver application by users. MachSMT is not a replacement for SMT solvers by any means. Instead, it is a tool that enables users to leverage the collective strength of the diverse set of algorithms implemented as part of these sophisticated solvers.

show abstract

Section: Algorithm Selection For Logic Solvers and Their Applicationsmentioning

confidence: 99%

MachSMT: A Machine Learning-based Algorithm Selector for SMT Solvers

Scott

Niemetz

Preiner

et al. 2021

Tools and Algorithms for the Construction and Analysis of Systems

View full text Add to dashboard Cite

show abstract

“…We achieve similar performance to MachSMT without pre-training by decoupling solver choice from time allocation and allowing for mistakes by selecting a sequence of solvers to run, instead of a single solver. Where4 [18] is a portfolio-based SMT solver that uses regression models to select which solver to run. It extracts features from WhyML programs rather than SMT queries and does not allocate time between solvers.…”

Section: Related Workmentioning

confidence: 99%

MedleySolver: Online SMT Algorithm Selection

Pimpalkhare

Mora

Polgreen

et al. 2021

Theory and Applications of Satisfiability Testing – SAT 2021

View full text Add to dashboard Cite

Satisfiability modulo theories (SMT) solvers implement a wide range of optimizations that are often tailored to a particular class of problems, and that differ significantly between solvers. As a result, one solver may solve a query quickly while another might be flummoxed completely. Predicting the performance of a given solver is difficult for users of SMT-driven applications, particularly when the problems they have to solve do not fall neatly into a well-understood category. In this paper, we propose an online algorithm selection framework for SMT called MedleySolver that predicts the relative performances of a set of SMT solvers on a given query, distributes time amongst the solvers, and deploys the solvers in sequence until a solution is obtained. We evaluate MedleySolver against the best available alternative, an offline learning technique, in terms of pure performance and practical usability for a typical SMT user. We find that with no prior training, MedleySolver solves 93.9% of the queries solved by the virtual best solver selector achieving 59.8% of the par-2 score of the most successful individual solver, which solves 87.3%. For comparison, the best available alternative takes longer to train than MedleySolver takes to solve our entire set of 2000 queries.

show abstract

“…SATzilla [34] considers SAT solver selection as an algorithm selection problem and adopts existing portfolio-solving techniques. Healy et al [19] apply similar techniques to the SMT solving domain. Our work focuses on how to apply portfolio-solving in the symbolic execution domain, and proposes domain-specific optimizations to accelerate the extraction of path constraint features.…”

Section: Background and Related Workmentioning

confidence: 99%

“…For example, KLEE [10] and JDART [25] embed the pySMT framework [15] and the JConstraints interface, respectively, as a unified interface for solvers. In addition, because solver selection is a type of algorithm selection problem, several portfolio-solving techniques have been applied to select solvers [19], [34]. However, because the time spent on constraint processing and prediction was not included in their evaluation, it remains unclear how effective these solutions are when applied to improve the overall performance of symbolic execution.…”

Section: Introductionmentioning

confidence: 99%

Enhancing Symbolic Execution by Machine Learning Based Solver Selection

Wen¹,

Mow²,

Chen³

et al. 2019

Proceedings 2019 Workshop on Binary Analysis Research

View full text Add to dashboard Cite

Constraint solving creates a serious performance bottleneck in symbolic execution. Examining a plethora of SMT solvers with diverse capabilities, we address the following research questions: How can the performance of symbolic execution improve if it can pick a priori the best solver for a given path constraint? How can such a prediction oracle be practically implemented? In this work, we first define the solver selection problem in symbolic execution and its evaluation metrics, and perform a preliminary study to gauge potential performance improvement through solver selection. We then present the design and implementation of Path Constraint Classifier (PCC), a machine learning based meta-solver that aims to reduce overall constraint solving latency by dynamically selecting a solver per query. Using machine learning seems straightforward, yet surprisingly underexplored; one main technical challenge is how to avoid excessive overhead introduced by feature extraction. We address this challenge by taking advantage of the structural characteristics of symbolic execution. Our experiments confirm that the overall solver time can be reduced by 10.3% in the KLEE dataset and 46% in the benchmark dataset, while the solver prediction procedure only accounts for 2% to 10% of overall solving time.

show abstract

Predicting SMT Solver Performance for Software Verification

Cited by 12 publications

References 32 publications

MachSMT: A Machine Learning-based Algorithm Selector for SMT Solvers

MachSMT: A Machine Learning-based Algorithm Selector for SMT Solvers

MedleySolver: Online SMT Algorithm Selection

Enhancing Symbolic Execution by Machine Learning Based Solver Selection

Contact Info

Product

Resources

About