Assessing Test Adequacy for Black-Box Systems without Specifications

Abstract

Abstract. Testing a black-box system without recourse to a specification is difficult, because there is no basis for estimating how many tests will be required, or to assess how complete a given test set is. Several researchers have noted that there is a duality between these testing problems and the problem of inductive inference (learning a model of a hidden system from a given set of examples). It is impossible to tell how many examples will be required to infer an accurate model, and there is no basis for telling how complete a given set of examples is. These issues have been addressed in the domain of inductive inference by developing statistical techniques, where the accuracy of an inferred model is subject to a tolerable degree of error. This paper explores the application of these techniques to assess test sets of black-box systems. It shows how they can be used to reason in a statistically justified manner about the number of tests required to fully exercise a system without a specification, and how to provide a valid adequacy measure for black-box test sets in an applied context.

Black-Box Test Generation from Inferred Models

“…This is useful because it provides a non code-based perspective on what software behaviour has (probably) been tested. This can therefore be used to formulate novel notions of test adequacy Weyuker [21], [18].…”

“…Until the mid-nineties this work was predominantly theoretical [3], [21], [4], [15], [14], [23], [22]. This developed strong theoretical underpinnings for test adequacy in an inference context (by linking to Machine-Learning notions such as Probably Approximately Correct learning, for example [23], [18]). …”

Behaviourally Adequate Software Testing

“…Given some element x (a given combination of inputs and outputs), c(x) = 0 or 1, depending on whether it belongs to the target concept (conforms to the behaviour of Figure 2. PAC-driven test adequacy assessment [4] the software system or not). The conventional assumption in PAC is that there exists some selection procedure EX(c,D) that randomly selects elements in X following some static distribution D (we do not need to know this distribution, but it must not change).…”

“…Several authors have attempted to use it in a purely theoretical setting to reason about "testability", or to reformulate syntax-based adequacy axioms [6], [7], [10]. More recently, Walkinshaw [4] showed how the PAC framework could be incorporated into a more general, statistically sound basis for assessing test set adequacy and to place bounds on the number of tests required to produce an adequate test set.…”

“…Previous work [4] thus assumed sufficiently large test sets, such that there is a high probability that all behaviour is triggered. However, when we are testing without a formal specification we are interested in producing small test sets, such that the developer can manually assess test outcomes, or can add test oracles in terms of test assertions.…”