Techniques for improving regression testing in continuous integration development environments

Abstract

In continuous integration development environments, software engineers frequently integrate new or changed code with the mainline codebase. This can reduce the amount of code rework that is needed as systems evolve and speed up development time. While continuous integration processes traditionally require that extensive testing be performed following the actual submission of code to the codebase, it is also important to ensure that enough testing is performed prior to code submission to avoid breaking builds and delaying the fast feedback that makes continuous integration desirable. In this work, we present algorithms that make continuous integration processes more cost-effective. In an initial presubmit phase of testing, developers specify modules to be tested, and we use regression test selection techniques to select a subset of the test suites for those modules that render that phase more costeffective. In a subsequent post-submit phase of testing, where dependent modules as well as changed modules are tested, we use test case prioritization techniques to ensure that failures are reported more quickly. In both cases, the techniques we utilize are novel, involving algorithms that are relatively inexpensive and do not rely on code coverage information -two requirements for conducting testing cost-effectively in this context. To evaluate our approach, we conducted an empirical study on a large data set from Google that we make publicly available. The results of our study show that our selection and prioritization techniques can each lead to costeffectiveness improvements in the continuous integration process.

The impact of continuous integration on other software development practices: A large-scale empirical study

“…Campos et al [48] enhanced CI with automated test generation. Elbaum et al [49] and Hu et al [50] applied test case selection and test case prioritization to test suites. Dösinger et al [51] proposed the continuous change impact analysis technique to improve the effectiveness of automated testing.…”

Tricorder: Building a Program Analysis Ecosystem

“…Because code reuse is common, most code depends upon a core set of libraries, and making changes to these base libraries may impact many projects. To ensure that changes do not break other projects, Google has a strong testing culture backed by continuous testing infrastructure [43], [15].…”

Experience Report: Automated System Level Regression Test Prioritization Using Multiple Factors

“…Elbaum et al [2] note that traditional regression testing techniques that rely on source code instrumentation and availability of a complete test set, become too expensive in continuous integration development environments, partly because the high frequency of code changes makes code coverage metrics imprecise and obsolete. Thus recent regression test techniques are based on information sources that are more readily available and light-weight.…”

“…Elbaum et al [2] use a simple prioritization of test suites based on their prior failure in a given time window. Their results show large variance in performance across window sizes, but typically better than no-prioritization.…”