Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact

Abstract

Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has and can be expected to have.

Covrig: a framework for the analysis of code, test, and coverage evolution in real software

“…In terms of similar infrastructure efforts, SIR [7] is a well-known repository for software testing research, which offers a variety of programs written in several different languages, together with test suites, bug data, and scripts. SIR also provides multiple versions for the same application, but typically less than a dozen.…”

RUGRAT: Runtime Test Case Generation Using Dynamic Compilers

“…We chose programs that use the most function pointers and/or contain the most error handling code. The set includes programs from the SPEC [22] and MiBench [13] suites, as well as the space application [11]. Table 2 lists the applications and their characteristics.…”

“…We performed a failure detection case study using the space [11] program to examine RU-GRAT's effectiveness at revealing faults and to assess how much tester effort RUGRAT can save. The space program includes extensive error handling code to handle application errors and over 13,000 test cases that are designed to maximize statement coverage.…”

Adaptive Regression Testing Strategy: An Empirical Study

“…We considered five Java programs obtained from the SIR infrastructure [28] as our objects of analysis: ant, xmlsecurity, jmeter, nanoxml, and galileo. Ant is a Java-based build tool, jmeter is a load testing tool for client/server application, and xml-security provides security functionality for XML data.…”

“…To control this threat, we validated our AHP tool using several examples. Other tools were from SIR [28], and they have been validated through numerous experiments. External Validity: Three issues limit the generalization of our results.…”