Prioritizing test cases for regression testing

Abstract

Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal. Various goals are possible; one involves rate of fault detection -a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test and let software engineers begin correcting faults earlier than might otherwise be possible. One application of prioritization techniques involves regression testing -the retesting of software following modifications; in this context, prioritization techniques can take advantage of information gathered about the previous execution of test cases to obtain test case orderings. In this paper, we describe several techniques for using test execution information to prioritize test cases for regression testing, including: (1) techniques that order test cases based on their total coverage of code components; (2) techniques that order test cases based on their coverage of code components not previously covered; (3) techniques that order test cases based on their estimated ability to reveal faults in the code components that they cover. We report the results of several experiments in which we applied these techniques to various test suites for various programs and measured the rates of fault detection achieved by the prioritized test suites, comparing those rates to the rates achieved by untreated, randomly ordered, and optimally ordered suites. Analysis of the data shows that each of the prioritization techniques studied improved the rate of fault detection of test suites, and this improvement occurred even with the least expensive of those techniques. The data also shows, however, that considerable room remains for improvement. The studies highlight several cost-benefits tradeoffs among the techniques studied, as well as several opportunities for future work.

Coverage-Based Test Cases Selection for XACML Policies

“…The policy target (lines [5][6][7][8][9][10][11][12] says that this policy applies to any subject, any action, any environment and the "books" resource. This policy has a first rule (ruleA) (lines 13-34) with a target (lines [14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33] specifying that this rule applies only to the access requests of a "read" action of "books", and "documents" resources with any environment. The effect of the second rule (ruleB) (lines 35-50) is Permit when the subject is "Julius", the action is "write", the resource and environment are any resource and any environment respectively.…”

“…Differently from this work, our proposal does not target regression systems and does not require the execution of test cases against the security policy for selecting test cases, then reducing cost and time effort of the overall testing process. Code coverage criteria are also addressed by most of the techniques of test cases prioritization [19] [20] [21], with the aim to reorder test cases so that those tests that have a higher priority are executed before the ones having a lower priority [22], [23]. Differently from these works, the target of our proposal is not test cases prioritization but test cases selection, i.e.…”

Test generation to expose changes in evolving programs

“…One stream of work has espoused test selection [2,12] -selecting a subset of tests from existing testsuite for running on the modified program. Another stream of work proposes test prioritization [4,15] -ordering tests in an existing test-suite to better meet testing objectives of the changed program.…”

“…Most of the past research efforts in regression testing focus on this inefficiency issue and provide solutions via test selection [2,12] (selecting a subset of the tests to be run) or test prioritization [4,15] (changing the order in which a set of given tests is run).…”

Adaptive Regression Testing Strategy: An Empirical Study

“…For instance, regression test selection techniques (e.g., [1], [2], [3]) reduce testing costs by selecting test cases that are necessary to test a modified program. Test case prioritization techniques (e.g., [4], [5], [6]) reorder test cases, scheduling test cases with the highest priority according to some criterion earlier in the testing process to yield benefits such as providing earlier feedback to testers and earlier fault detection.…”