Assessing and generating test sets in terms of behavioural adequacy

Abstract

SUMMARYIdentifying a finite test set that adequately captures the essential behaviour of a program such that all faults are identified is a well-established problem. This is traditionally addressed with syntactic adequacy metrics (e.g. branch coverage), but these can be impractical and may be misleading even if they are satisfied. One intuitive notion of adequacy, which has been discussed in theoretical terms over the past three decades, is the idea of behavioural coverage: If it is possible to infer an accurate model of a system from its test executions, then the test set can be deemed to be adequate. Despite its intuitive basis, it has remained almost entirely in the theoretical domain because inferred models have been expected to be exact (generally an infeasible task) and have not allowed for any pragmatic interim measures of adequacy to guide test set generation. This paper presents a practical approach to incorporate behavioural coverage. Our BESTEST approach (1) enables the use of machine learning algorithms to augment standard syntactic testing approaches and (2) shows how search-based testing techniques can be applied to generate test sets with respect to this criterion. An empirical study on a selection of Java units demonstrates that test sets with higher behavioural coverage significantly outperform current baseline test criteria in terms of detected faults.

Empirical study of correlation between mutation score and model inference based test suite adequacy assessment

“…The authors also use the term adequacy, which was defined in [3] where a test suite is adequate if it implies no errors in the SUT if it executes correctly. In [2] the authors do not obtain a positive correlation at all for their examples with categorical outcomes when using a similar machine learning based test suite adequacy assessment. The approach introduced in this work to assess the adequacy of a test suite by model inference is similar to the idea of probably approximately correct (PAC) learning [15].…”

“…In their recent work Fraser and Walkinshaw [2] answer the question whether there exists a relationship between their adequacy score and the ability of a test suite to detect defects. The authors also use the term adequacy, which was defined in [3] where a test suite is adequate if it implies no errors in the SUT if it executes correctly.…”

“…BMI: The body mass index (BMI) example is a program that categorizes a person by means of weight and height into one of five different categories. For this example we used the source code published in [2].…”

Inferring Computational State Machine Models from Program Executions

“…In this section we provide a small case study to (1) show that the approach can be applied to reverse-engineer Java APIs for larger frameworks, and (2) provide an illustration of how inferred models can be used for regression testing, as in the vein of growing body of work [1], [2], [33]: We use the inferred functions to derive assertions that can be used as oracles, and illustrate how the model can be used to identify test-inputs that explore previously unexplored behaviour.…”

“…For example, x = 9.731, y = 13.606 (which will appear if the test case in Figure 6 is executed) contradict the assertion. In a typical inference-driven testing cycle [33], [36], these new inputs would be assimilated into the test set, and the model would be inferred afresh, repeating the cycle until no contradictory inputs can be found.…”