Lightweight Automated Testing with Adaptation-Based Programming

Kulesza

IIEEE Trans. Software Eng.

et al. 2014

Self Cite

Abstract-How do you test a program when only a single user, with no expertise in software testing, is able to determine if the program is performing correctly? Such programs are common today in the form of machine-learned classifiers. We consider the problem of testing this common kind of machine-generated program when the only oracle is an end user : e.g., only you can determine if your email is properly filed. We present test selection methods that provide very good failure rates even for small test suites, and show that these methods work in both large-scale random experiments using a "gold standard" and in studies with real users. Our methods are inexpensive and largely algorithm-independent. Key to our methods is an exploitation of properties of classifiers that is not possible in traditional software testing. Our results suggest that it is plausible for time-pressured end users to interactively detect failures-even very hard-to-find failures-without wading through a large number of successful (and thus less useful) tests. We additionally show that some methods are able to find the arguably most difficult-to-detect faults of classifiers: cases where machine learning algorithms have high confidence in an incorrect result.

Section: Active Learningmentioning

confidence: 99%

You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems

Kulesza

IIEEE Trans. Software Eng.

et al. 2014

Self Cite

“…Finally, only branch and statement coverage are widely enough implemented for languages that it is safe to assume anyone interested in producing a quick test has tools to support their use. For random testing, which is often carried out by developers or by security experts, this last condition is important: lightweight methods that do not require static or dynamic analysis expertise and are easy to implement from scratch are more likely to be widely applied [24]. …”

Section: A Coverage As An Effectmentioning

confidence: 99%

Cause Reduction for Quick Testing

Alipour

2014 IEEE Seventh International Conference on Software Testing, Verification and Validation

et al. 2014

Self Cite

Abstract-In random testing, it is often desirable to produce a "quick test" -an extremely inexpensive test suite that can serve as a frequently applied regression and allow the benefits of random testing to be obtained even in very slow or oversubscribed test environments. Delta debugging is an algorithm that, given a failing test case, produces a smaller test case that also fails, and typically executes much more quickly. Delta debugging of random tests can produce effective regression suites for previously detected faults, but such suites often have little power for detecting new faults, and in some cases provide poor code coverage. This paper proposes extending delta debugging by simplifying tests with respect to code coverage, an instance of a generalization of delta debugging we call cause reduction. We show that test suites reduced in this fashion can provide very effective quick tests for real-world programs. For Mozilla's SpiderMonkey JavaScript engine, the reduced suite is more effective for finding software faults, even if its reduced runtime is not considered. The effectiveness of a reduction-based quick test persists through major changes to the software under test.

“…6) Java Container Classes: The final set of subjects are taken from Java container classes used in multiple papers to compare test generation methods [15], [16]. These all have very small APIs (3 methods -insert, remove, and find).…”

Section: A Experimental Subjectsmentioning

confidence: 99%

“…For these subjects, the only features are remove and find, as tests with no insert operations are essentially identical (as state never changes). One possible explanation of the previous inattention to suppression effects may be that suppression is less problematic for simple container classes, which have been the subjects of many studies [15], [16], [17]. These subjects were easily covered by ∼5,000 length-200 tests each.…”

Section: A Experimental Subjectsmentioning

confidence: 99%

Help, help, i'm being suppressed! The significance of suppressors in software testing

2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE)

Alipour

et al. 2013

Self Cite

Test features are basic compositional units used to describe what a test does (and does not) involve. For example, in API-based testing, the most obvious features are function calls; in grammar-based testing, the obvious features are the elements of the grammar. The relationship between features as abstractions of tests and produced behaviors of the tested program is surprisingly poorly understood. This paper shows how large-scale random testing modified to use diverse feature sets can uncover causal relationships between what a test contains and what the program being tested does. We introduce a general notion of observable behaviors as targets, where a target can be a detected fault, an executed branch or statement, or a complex coverage entity such as a state, predicate-valuation, or program path. While it is obvious that targets have triggers -features without which they cannot be hit by a testthe notion of suppressors -features which make a test less likely to hit a target -has received little attention despite having important implications for automated test generation and program understanding. For a set of subjects including C compilers, a flash file system, and JavaScript engines, we show that suppression is both common and important.