An Information Retrieval Approach for Regression Test Prioritization Based on Program Changes

Abstract

Regression testing is widely used in practice for validating program changes. However, running large regression suites can be costly. Researchers have developed several techniques for prioritizing tests such that the higher-priority tests have a higher likelihood of finding bugs. A vast majority of these techniques are based on dynamic analysis, which can be precise but can also have significant overhead (e.g., for program instrumentation and test-coverage collection). We introduce a new approach, REPiR, to address the problem of regression test prioritization by reducing it to a standard Information Retrieval problem such that the differences between two program versions form the query and the tests constitute the document collection. REPiR does not require any dynamic profiling or static program analysis. As an enabling technology we leverage the open-source IR toolkit Indri. An empirical evaluation using eight open-source Java projects shows that REPiR is computationally efficient and performs better than existing (dynamic or static) techniques for the majority of subject systems.

Reinforcement learning for automatic test case prioritization and selection in continuous integration

“…Strandberg et al [35] apply a novel prioritization method with multiple factors in a real-world embedded software and show the improvement over industry practice. Other regression test selection techniques have been proposed based on historical test data [16,19,23,25], code dependencies [14], or information retrieval [17,33]. Despite various approaches to test optimization for regression testing, the challenge of applying most of them in practice lies in their complexity and the computational overhead typically required to collect and analyze di erent test parameters needed for prioritization, such as age, test coverage, etc.…”

Automatically Finding Hidden Industrial Criteria used in Test Selection

“…This suggests investigating additional criteria that can be used in practice. For instance, information retrieval [7] also uses ranking functions based on keyword indexing and frequency. We intend to consider this new criterion as one of our future work.…”

Test-case prioritization: achievements and challenges

“…However, these search-based algorithms are inferior without significant difference to the additional strategy in terms of APXC, which measures the average percentage of structural coverage of the prioritized test suite. Besides, Saha et al [44] presented a new approach REPiR which transfers the problem of test-case prioritization to a standard information retrieval problem. Tonella et al [45] presented a machine learning based approach that incorporates user knowledge into test-case prioritization.…”