Behaviourally Adequate Software Testing

Abstract

Abstract-Identifying a finite test set that adequately captures the essential behaviour of a program such that all faults are identified is a well-established problem. Traditional adequacy metrics can be impractical, and may be misleading even if they are satisfied. One intuitive notion of adequacy, which has been discussed in theoretical terms over the past three decades, is the idea of behavioural coverage; if it is possible to infer an accurate model of a system from its test executions, then the test set must be adequate. Despite its intuitive basis, it has remained almost entirely in the theoretical domain because inferred models have been expected to be exact (generally an infeasible task), and have not allowed for any pragmatic interim measures of adequacy to guide test set generation. In this work we present a new test generation technique that is founded on behavioural adequacy, which combines a model evaluation framework from the domain of statistical learning theory with search-based white-box test generation strategies. Experiments with our BESTEST prototype indicate that such test sets not only come with a statistically valid measurement of adequacy, but also detect significantly more defects.

Inferring extended finite state machine models from software executions

“…So far, this has concentrated solely on simple finite state machines [48,49,42] and data classifiers [18]. It is envisaged that the ability to incorporate these richer models will lead the the ability to produce more more rigorous test sets.…”

“…Their use is becoming increasingly popular from a testing perspective, where they have been used as oracles for regression testing [17], to detect intrusions and anomalous behaviour [43], and to automatically generate test cases [18,42]. Beyond testing, inferred models can aid software comprehension [12] and requirements engineering [14].…”

Black-Box Test Generation from Inferred Models

“…Ghani and Clark [8] presented a technique to refine reverse-engineered Daikon models by finding test cases to contradict the models. Fraser and Walkinshaw [7] presented a model-inference driven framework that follows the same loop, and uses the same algorithms as those presented here. However, crucially, their approach is not black-box; the core test-sets are generated by conventional white-box syntax coverage.…”

“…This was previously used in previous work by the authors [7]. We selected a quasi-random initial test set (with the inputs constrained to achieve all of the possible body-mass categories at least once).…”

“…[8], [7], or to use state machine testing strategies [13], [20], [19], [11]. 5) The loop is repeated, until some termination criterion is met (e.g.…”