How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub Projects

Abstract

Test Case Prioritization (TCP) is an increasingly important regression testing technique for reordering test cases according to a pre-defined goal, particularly as agile practices gain adoption. To better understand these techniques, we perform the first extensive study aimed at empirically evaluating four static TCP techniques, comparing them with state-of-research dynamic TCP techniques across several quality metrics. This study was performed on 58 real-word Java programs encompassing 714 KLoC and results in several notable observations. First, our results across two effectiveness metrics (the Average Percentage of Faults Detected APFD and the cost cognizant APFDc) illustrate that at test-class granularity, these metrics tend to correlate, but this correlation does not hold at test-method granularity. Second, our analysis shows that static techniques can be surprisingly effective, particularly when measured by APFDc. Third, we found that TCP techniques tend to perform better on larger programs, but that program size does not affect comparative performance measures between techniques. Fourth, software evolution does not significantly impact comparative performance results between TCP techniques. Fifth, neither the number nor type of mutants utilized dramatically impact measures of TCP effectiveness under typical experimental settings. Finally, our similarity analysis illustrates that highly prioritized test cases tend to uncover dissimilar faults. and his thesis focused on improving bug reporting for mobile apps through novel applications of program analysis techniques. He has published in several top peerreviewed software engineering venues including: ICSE, ESEC/FSE, IC-SME, ICST, and MSR. He was recently recognized as the second-overall winner among graduate students in the ACM Student Research competition at ESEC/FSE15. Moran is a student member of IEEE and ACM and has served as an external reviewer for ICSE, ICSME, FSE, APSEC, and SCAM. More information available at https://www.kpmoran.com.

Regression test case prioritization by code combinations coverage

“…The additional strategy also selects test cases according to a descending order, but uses the number of code units not already covered by previously selected test cases. According to previous studies [17,18,19], although seemingly simple, the greedy strategies (especially additional) perform better than most other RTCP techniques in terms of the fault detection rate. Therefore, in our study, we used a simple greedy algorithm to instantiate the CCCP prioritization function for statement, branch, and method coverage criteria.…”

“…Finally, the search-based technique considers all permutations as candidate solutions, and uses a meta-heuristic search algorithm to guide the search for a better test execution order -a genetic algorithm was adopted in this study, due to its effectiveness [2]. These four test prioritization techniques, whose details are presented in Table 1, have been widely used in RTCP previous studies [17,18,20].…”

“…Mutation testing has been argued to be an appropriate approach for assessing fault detection performance [27,28,29]. Mutation testing has also been used in recent RTCP research studies [17,18,19,20]. However, Luo et al [49] has highlighted the differences between real faults and mutants, explaining that the relative performances of RTCP techniques on mutants may not translate to similar relative performances with real faults.…”

“…Citing a lack of comprehensive studies comparing static and dynamic test prioritization techniques, Luo et al [17,18] compared static RTCP techniques with dynamic ones. Henard et al [19] compared white-box and back-box RTCP techniques.…”

A Regression Test Based on the Test Case Prioritization Techniques by using the Nature of Bees

“…Regression testing is a process/methodology to effectively and efficiently validates the errors after doing some changes to the existing software [9][10 [11]. It is also called as validation test that which leads to assure the [24].…”