Cause Reduction for Quick Testing

Abstract

Abstract-In random testing, it is often desirable to produce a "quick test" -an extremely inexpensive test suite that can serve as a frequently applied regression and allow the benefits of random testing to be obtained even in very slow or oversubscribed test environments. Delta debugging is an algorithm that, given a failing test case, produces a smaller test case that also fails, and typically executes much more quickly. Delta debugging of random tests can produce effective regression suites for previously detected faults, but such suites often have little power for detecting new faults, and in some cases provide poor code coverage. This paper proposes extending delta debugging by simplifying tests with respect to code coverage, an instance of a generalization of delta debugging we call cause reduction. We show that test suites reduced in this fashion can provide very effective quick tests for real-world programs. For Mozilla's SpiderMonkey JavaScript engine, the reduced suite is more effective for finding software faults, even if its reduced runtime is not considered. The effectiveness of a reduction-based quick test persists through major changes to the software under test.

Using test case reduction and prioritization to improve symbolic execution

“…In all cases, we used statement coverage-based reduction, even though symbolic execution is based on branches. Statement coverage-based reduction in practice is much faster than branch coverage-based reduction, but tends to preserve branch coverage well in the final result [12].…”

“…the YAFFS2 file system or a container class), we can simply use each API call as a component and run one reduction. Note that while we count cause reduction against test budgets in our experiments, in practice projects might maintain cause reduced tests as fast regressions ("quick tests") [12]. In all cases, we used statement coverage-based reduction, even though symbolic execution is based on branches.…”

“…Cause reduction uses the usual ddmin algorithm of delta debugging, but replaces the check that reduced tests fail with a check that a reduced test covers at least all statements covered by the original test case. In experiments with the SpiderMonkey JavaScript engine, a suite of tests where each test was reduced while preserving statement coverage surprisingly showed better fault detection than the original suite, and statement coverage-based reduction was shown to preserve other important properties well (test case failure, branch coverage, and mutation kills) [12].…”

“…Therefore, given two test cases with the same code coverage, it seems likely that symbolic exploration based on the "smaller" test case will be more effective, though it is possible that the change could reduce the value of the seed test case (for example constraints may change and branch feasibility in context may change). Cause reduction allows us, given a test case T to use delta-debugging [16] to produce a new test case T ′ such that T and T ′ have the same code coverage, and removing any component of T ′ will result in lower code coverage [12]. Cause reduction uses the usual ddmin algorithm of delta debugging, but replaces the check that reduced tests fail with a check that a reduced test covers at least all statements covered by the original test case.…”

“…Delta-debugging [33] is a well-known technique for reducing the size of test cases, but it is only useful for failing test cases, and often reduces test cases so much that they are a poor basis for symbolic exploration. However, recent work has shown that delta-debugging using preservation of source code coverage rather than failure as a property to be preserved can produce extremely efficient regression suites based on existing tests [12].…”