Regression Test Selection for Distributed Software Histories

Abstract

Regression test selection analyzes incremental changes to a codebase and chooses to run only those tests whose behavior may be affected by the latest changes in the code. By focusing on a small subset of all the tests, the testing process runs faster and can be more tightly integrated into the development process. Existing techniques for regression test selection consider two versions of the code at a time, effectively assuming a development process where changes to the code occur in a linear sequence.Modern development processes that use distributed version-control systems are more complex. Software version histories are generally modeled as directed graphs; in addition to version changes occurring linearly, multiple versions can be related by other commands, e.g., branch, merge, rebase, cherry-pick, revert, etc. This paper describes a regression test-selection technique for software developed using modern distributed version-control systems. By modeling different branch or merge commands directly in our technique, it computes safe test sets that can be substantially smaller than applying previous techniques to a linearization of the software history.We evaluate our technique on software histories of several large opensource projects. The results are encouraging: our technique obtained an average of 10.89× reduction in the number of tests over an existing technique while still selecting all tests whose behavior may differ. 1 We use the term "version" for what version-control systems often call "revision".

An empirical evaluation and comparison of manual and automated test selection

“…One key difference among these techniques is the granularity level at which they detect affected tests (e.g., statement, controlflow edge, method, etc.). Recent work [10,16,34,37,38] has also reported some success with automated RTS in large software projects. Unfortunately, after decades of research, no practical RTS tool is available for widespread adoption by developers.…”

“…9 Version 7.4, build id: 201310111528. 10 We selected VisualStudio 2010 rather than the latest version because VisualStudio 2010 was the only IDE that has ever supported automated RTS; interestingly enough, this automated RTS support has been removed from the IDE in subsequent releases!…”

SPIRITuS: a SimPle Information Retrieval regressIon Test Selection approach

“…Safe RTS approaches guarantee by construction that all test cases that can reveal faults are selected. Examples of safe approaches are those proposed by Rothermel and Harrold [31], Vokolos and Frank [15], and Gligoric et al [16,32]. Rothermel and Harrold's approach build a graph-based representation (i.e., control-flow-graph) of P and P′.…”

Mining performance regression inducing code changes in evolving software

“…The default approach for regression testing is to retest all test cases after releasing a new version, which is an expensive proposition. To solve this problem, a number of techniques for selecting regression tests have been proposed [26,49,21,66,32,74]. Grosso et al proposed an approach that uses GAs to generate test cases that cause buffer overflows and integrate domain knowledge with slicing and static analysis to reduce the search space [22].…”