Selecting a Cost-Effective Test Case Prioritization Technique

Abstract

Regression testing is an expensive testing process used to validate modified software and detect whether new faults have been introduced into previously tested code. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One goal of prioritization is to increase a test suite's rate of fault detection. Previous empirical studies have shown that several prioritization techniques can significantly improve rate of fault detection, but these studies have also shown that the effectiveness of these techniques varies considerably across various attributes of the program, test suites, and modifications being considered. This variation makes it difficult for a practitioner to choose an appropriate prioritization technique for a given testing scenario. To address this problem, we analyze the fault detection rates that result from applying several different prioritization techniques to several programs and modified versions. The results of our analyses provide insights into what types of prioritization techniques are and are not appropriate under specific testing scenarios, and the conditions under which they are or are not appropriate. Our analysis approach can also be used by other researchers or practitioners to determine the prioritization techniques appropriate to other workloads.

Improving Regression Testing Transparency and Efficiency with History-Based Prioritization -- An Industrial Case Study

“…In contrast, most prioritization techniques are not depending on knowledge about modifications [4] but are instead based on supposed equivalents for high fault detection rates (e.g early code coverage, test suite coverage over a number of sessions or historical fault detection effectiveness). Elbaum et al investigate different coverage criteria that may govern the prioritization [13]. Kim and Porter emphasize the need to view RT as an ordered sequence of test sessions, where the history of the test case executions should be considered [14].…”

“…The metrics used for comparing the techniques are accepted by the research community and have been used in several previous studies for the same purposes [22], [7], [13]. There is however other views on what is a good selection (e.g.…”

A bat-inspired algorithm for prioritizing test cases

“…In addition, possible count of orders of a test suite T s generated from P is as much as n factorial. The success of the ordering can be evaluated not only with code coverage and model coverage, but also with some measurement methods such as APFD [14] and APXC [11]. If a search-based optimization is preferred, the code coverage criteria is suitable.…”

Adaptive Regression Testing Strategy: An Empirical Study

“…However, studies to date have not considered strategies for selecting appropriate techniques under particular circumstances as systems evolve. Only few studies [19], [20] have done on the problem of helping practitioners choose appropriate techniques under particular system and process constraints. Harrold et al [20] present empirical results that demonstrate how code modifications can affect the choice of regression test selection methods.…”

“…Harrold et al [20] present empirical results that demonstrate how code modifications can affect the choice of regression test selection methods. Elbaum et al [19] perform experiments exploring characteristics of program structure, test suite composition, and changes on prioritization, and identified several metrics characterizing these attributes that correlate with prioritization effectiveness. The empirical results of their study provide insights into which prioritization technique is appropriate (or not appropriate) under specific testing scenarios.…”