Evaluating program analysis and testing tools with the RUGRAT random benchmark application generator

Abstract

Benchmarks are heavily used in different areas of computer science to evaluate algorithms and tools. In program analysis and testing, open-source and commercial programs are routinely used as benchmarks to evaluate different aspects of algorithms and tools. Unfortunately, many of these programs are written by programmers who introduce different biases, not to mention that it is very difficult to find programs that can serve as benchmarks with high reproducibility of results.We propose a novel approach for generating random benchmarks for evaluating program analysis and testing tools. Our approach uses stochastic parse trees, where language grammar production rules are assigned probabilities that specify the frequencies with which instantiations of these rules will appear in the generated programs. We implemented our tool for Java and applied it to generate benchmarks with which we evaluated different program analysis and testing tools. Our tool was also implemented by a major software company for C++ and used by a team of developers to generate benchmarks that enabled them to reproduce a bug in less than four hours.

UNICORN: a unified approach for localizing non‐deadlock concurrency bugs

“…Threats to external validity limit the extent to which the results will generalize to other kinds of concurrent programs. Whereas there are large benchmark suites and benchmark generators for sequential programs, there is no large and widely accepted bug benchmark suites for concurrent programs, and thus this threat is common to all prior work in this area. To mitigate this problem, the benchmark suite includes most of the benchmarks used in other work and furthermore, includes both Java and C++ programs.…”

A Comparative Study of Incremental Constraint Solving Approaches in Symbolic Execution

“…A practical challenge for these kinds of generators is to construct realistic programs. However, an empirical study indicates that it is statistically impossible for a program analysis technique to differentiate a program written by a human from one that the tool generates [20]. The study compared real and generated programs with 78 existing software metrics.…”

RUGRAT: Evaluating program analysis and testing tools and compilers with large generated random benchmark applications

“…‡ ‡ This article is an extended version of our previous work presented at the 10th International Workshop on Dynamic Analysis (WODA 2012) .…”