Reducing the Costs of Bounded-Exhaustive Testing

Abstract

Abstract. Bounded-exhaustive testing is an automated testing methodology that checks the code under test for all inputs within given bounds: first the user describes a set of test inputs and provides test oracles that check test outputs; then the tool generates all the inputs, executes them on the code under test, and checks the outputs; and finally the user inspects failing tests to submit bug reports. The costs of boundedexhaustive testing include machine time for test generation and execution (which translates into human time waiting for these results) and human time for inspection of results. This paper proposes three techniques that reduce these costs. Sparse Test Generation skips some tests to reduce the time to the first failing test. Structural Test Merging generates a smaller number of larger test inputs (rather than a larger number of smaller test inputs) to reduce test generation and execution time. Oracle-based Test Clustering groups failing tests to reduce the inspection time. Results obtained from the bounded-exhaustive testing of the Eclipse refactoring engine show that these three techniques can substantially reduce the costs while mostly preserving fault-detection capability.

Test generation through programming in UDITA

“…To generate such programs, we need to both "generate inheritance graphs" and "add methods" in the classes and interfaces in the graphs. Our experience with UDITA's combined declarative/imperative style shows that, compared to our prior approach, ASTGen [19,35], UDITA is more expressive, resulting in shorter (and easier to write) test generation programs, and sometimes provides faster generation despite executing on the significantly slower JPF virtual machine. Through these experiments, we revealed new bugs in Eclipse and NetBeans, and even a bug in the Sun Java compiler.…”

“…In a purely declarative style, embodied in techniques such as TestEra [36] and Korat [7,22,29,41,42], the tester writes the predicates-what the test inputs should satisfy; then the tool searches for valid tests. In contrast, in purely imperative style, embodied in techniques such as ASTGen [19,35], the tester directly writes generators-how to generate valid inputs; then the tool executes these generators to generate the inputs. We first present these two pure approaches, then discuss how UDITA allows freely combining them, and finally how UDITA efficiently generates inputs.…”

“…The second extension is the object pool abstraction that allows the tester to control generation of linked structures with any desired sharing patterns, including trees but also DAGs, cyclic graphs, and domainspecific data structures. Due to its expressive power, UDITA enables testers to write test generation programs using any desirable mixture of declarative [7,22,29,36,41,42] and imperative [19,35] styles, whereas previous systems required the use of only one style.…”

Model-Based Filtering of Combinatorial Test Suites

“…In [15], authors propose to study test reduction in the context of boundedexhaustive testing, which could be described as a variation of combinatorial testing. Three techniques are proposed to reduce test generation, execution time and diagnosis.…”

Automatic test suite evolution

“…Therefore, the software developers need to inspect and understand the generated test cases to complete and execute them [JLDM09]. Model-based testing generates both test inputs and oracles from software specifications, and thus does not require developers to modify or complete the generated test cases [DNSVT07].…”