Probabilistic regression suites for functional verification

Abstract

Random test generators are often used to create regression suites on-the-fly. Regression suites are commonly generated by choosing several specifications and generating a number of tests from each one, without reasoning which specification should be used and how many tests should be generated from each specification. This paper describes a technique for building high quality random regression suites. The proposed technique uses information about the probability of each test specification covering each coverage task. This probability is used, in turn, to determine which test specifications should be included in the regression suite and how many tests should be generated from each specification. Experimental results show that this practical technique can be used to improve the quality, and reduce the cost, of regression suites. Moreover, it enables better informed decisions regarding the size and distribution of the regression suites, and the risk involved.

A probabilistic alternative to regression suites

“…Not surprisingly, our first attempts [9] were an adaptation of the known algorithms for test selection and prioritization to our domain. The main difference between the domain of these known algorithms and our domain is that in our domain a given set of input parameters impacts the probability of having certain characteristics in the tests.…”

“…The construction of a random regression suite can thus be expressed by the following optimization problem [9]:…”

“…Given a set of reliable probability estimates, we showed in [9] how the construction of efficient random regression suites can be formalized as an optimization problem. We addressed two variants of the problem.…”

“…The greedy algorithms suggested in [9], as well as all standard greedy algorithms, are aware of the past but not of the future. Indeed, algorithms that try to predict the future, usually using searches, are not considered greedy.…”

“…We also show that this algorithm is an improvement over the greedy algorithm for the classical set cover problem. We expanded the work done in [9] to the case where coverage is used to measure the utility of tests executed so far. This additional information enables us to get better results, but requires modification of the algorithms for on-the-fly selection.…”