MUX: algorithm selection for software model checkers

Abstract

With the growing complexity of modern day software, software model checking has become a critical technology for ensuring correctness of software. As is true with any promising technology, there are a number of tools for software model checking. However, their respective performance trade-offs are difficult to characterize accurately -making it difficult for practitioners to select a suitable tool for the task at hand. This paper proposes a technique called MUX that addresses the problem of selecting the most suitable software model checker for a given input instance. MUX performs machine learning on a repository of software verification instances. The algorithm selector, synthesized through machine learning, uses structural features from an input instance, comprising a program-property pair, at runtime and determines which tool to use.We have implemented MUX for Windows device drivers and evaluated it on a number of drivers and model checkers. Our results are promising in that the algorithm selector not only avoids a significant number of timeouts but also improves the total runtime by a large margin, compared to any individual model checker. It also outperforms a portfolio-based algorithm selector being used in Microsoft at present. Besides, MUX identifies structural features of programs that are key factors in determining performance of model checkers.

Predicting rankings of software verification tools

“…Our approach can be seen as a tool for algorithm selection, a problem that has also been tackled by other authors [6,23,27]. Other applications of machine learning include the learning of programs from examples ( [13,15]) and the prediction of properties of programs (e.g., types for program variables [16] or malware in Android apps [17]).…”

“…e simplest way of de ning a kernel is via the inner product of feature vectors, i.e., vectorial representations of data objects. In the two approaches existing so far [6,23], corresponding features of programs, such as the number of loops, conditionals, pointer variables, or arrays in a program, are de ned in an explicit way. Obviously, this approach requires su cient domain knowledge to identify those features that are important for the prediction problem at hand.…”

“…As training data, we take the SV-COMP results of 2015 and the only recently published results of 2017. e key ingredient of our approach is the de nition of a number of kernels [20], which act as similarity measures on veri cation tasks. So far, two other machine learning methods for selecting tools or algorithms for veri cation have been proposed [6,23], both of them being based on feature vectors: while Tulsian et al [23] only employ structural features of programs (like the number of arrays, loops, recursive functions), Demyanova et al [6] uses a number of data-ow analyses to also determine more sophisticated features (e.g., certain loop pa erns). us, both approaches try to explicitly capture aspects of source code that make veri cation hard (for some or all tools).…”

Empirical software metrics for benchmarking of verification tools

“…A machine-learning based method for selecting model checkers was previously introduced in [39]. Similar to our work, the authors use SVM classification with weights (cf.…”

“…We highlight the gold, silver, and bronze medal in dark gray, light gray and white + bold font, respectively. The last row shows the number of gold/silver/bronze medals won in individual categories [39]). 4.…”