Using machine learning to refine Category-Partition test specifications and test suites

2015 IEEE/ACM 4th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering

2015

Abstract-Automatically generating test inputs for components without source code (are 'black-box') and specification is challenging. One particularly interesting solution to this problem is to use Machine Learning algorithms to infer testable models from program executions in an iterative cycle. Although the idea has been around for over 30 years, there is little empirical information to inform the choice of suitable learning algorithms, or to show how good the resulting test sets are. This paper presents an openly available framework to facilitate experimentation in this area, and provides a proof-of-concept inference-driven testing framework, along with evidence of the efficacy of its test sets on three programs.

Section: A Methodologymentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Black-Box Test Generation from Inferred Models

Papadopoulos

2015 IEEE/ACM 4th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering

2015

“…Since then, the idea of combining model inference with software testing has been comprehensively explored in several theoretical and practical contexts [8], [6], [7], [9], [10], [11], [12], [13]. Much of this work has focussed on the appealing, complementary relationship between program testing and machine learning.…”

Section: A Current Approaches To Behavioural Adequacy and Their Limimentioning

confidence: 99%

Behaviourally Adequate Software Testing

2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

2012

Abstract-Identifying a finite test set that adequately captures the essential behaviour of a program such that all faults are identified is a well-established problem. Traditional adequacy metrics can be impractical, and may be misleading even if they are satisfied. One intuitive notion of adequacy, which has been discussed in theoretical terms over the past three decades, is the idea of behavioural coverage; if it is possible to infer an accurate model of a system from its test executions, then the test set must be adequate. Despite its intuitive basis, it has remained almost entirely in the theoretical domain because inferred models have been expected to be exact (generally an infeasible task), and have not allowed for any pragmatic interim measures of adequacy to guide test set generation. In this work we present a new test generation technique that is founded on behavioural adequacy, which combines a model evaluation framework from the domain of statistical learning theory with search-based white-box test generation strategies. Experiments with our BESTEST prototype indicate that such test sets not only come with a statistically valid measurement of adequacy, but also detect significantly more defects.

“…Initially, Weyuker's work and subsequent work by Bergadano et al [3,30] focussed on synthesised programs. Since then however, similar approaches have been based upon Artificial Neural Nets [12,22], invariants [8], decision trees [5] and deterministic finite state automata [2,19,23,26,28,29].…”

Section: Testing With Inductive Inferencementioning

confidence: 99%

“…There has been a recent resurgence in techniques that expolit this relationship [3,5,6,8,12,17,19,22,23,26,28,29,30] by inferring models from test sets, and in some cases using these models to elicit further test cases. However, these techniques tend to suffer from two problems: (1) there is no means of predicting how many tests would be required to arrive at an adequate test set and (2) given a partial test set, there is no basis for gauging how close it is to being adequate.…”

Section: Introductionmentioning

confidence: 99%

Assessing Test Adequacy for Black-Box Systems without Specifications

Testing Software and Systems

2011

Abstract. Testing a black-box system without recourse to a specification is difficult, because there is no basis for estimating how many tests will be required, or to assess how complete a given test set is. Several researchers have noted that there is a duality between these testing problems and the problem of inductive inference (learning a model of a hidden system from a given set of examples). It is impossible to tell how many examples will be required to infer an accurate model, and there is no basis for telling how complete a given set of examples is. These issues have been addressed in the domain of inductive inference by developing statistical techniques, where the accuracy of an inferred model is subject to a tolerable degree of error. This paper explores the application of these techniques to assess test sets of black-box systems. It shows how they can be used to reason in a statistically justified manner about the number of tests required to fully exercise a system without a specification, and how to provide a valid adequacy measure for black-box test sets in an applied context.