Test case prioritization: a family of empirical studies

Abstract

To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of such prioritization is to increase a test suite's rate of fault detection. Previous work reported the results of studies that showed that prioritization techniques can signi cantly improve rate of fault detection. Those studies, however, raised several additional questions: 1 can prioritization techniques be e ective when targeted at speci c modi ed versions; 2 what tradeo s exist between ne granularity and coarse granularity prioritization techniques; 3 can the incorporation of measures of fault proneness into prioritization techniques improve their e ectiveness? To address these questions, we have performed several new studies, in which we empirically compared prioritization techniques using both controlled experiments and case studies. The results of these studies show that each of the prioritization techniques considered can improve the rate of fault detection of test suites overall. Finegranularity techniques typically outperformed coarse-granularity techniques, but only by a relatively small margin overall; in other words, the relative imprecision in coarse-granularity analysis did not dramatically reduce their ability to improve rate of fault detection. Incorporation of fault-proneness techniques produced relatively small improvements over other techniques in terms rate of fault detection, a result which ran contrary to our expectations. Our studies also show that the relative e ectiveness of various techniques can vary signi cantly across target programs. Furthermore, our analysis shows that whether the e ectiveness di erences observed will result in savings in practice varies substantially with the cost factors associated with particular testing processes. Further work to understand the sources of this variance, and to incorporate such understanding into prioritization techniques and the choice of techniques, would be bene cial.

Towards Regression Testing Constraints

“…It has also confirmed the previous findings derived from different empirical studies [3], [4], [14]. Thus, the need for a prioritization technique selection approach is of considerable importance to the testers' community.…”

“…At the end of the subject program analysis, we ended up with a 41 versions for each base program, variant number of test cases with the specified code coverage criteria [4]. Test Pools were generated using a combination of white-box/black-box techniques focusing on coverage criteria.…”

Kỹ Thuật Kiểm Thử Hồi Qui Hiệu Quả Cho Phát Triển Ứng Dụng Di Động

“…Có 18 kỹ thuật ưu tiên hóa các ca kiểm thử được xác định ở [23], như các kỹ thuật so sánh, các kỹ thuật mức dòng lệnh và các kỹ thuật mức chức năng. Hiện tại, có nhiều thuật toán tìm độ ưu tiên ca kiểm thử được phát triển bởi nhiều nhà nghiên cứu trong lĩnh vực này.…”

Automating Test Case Design within the Classification Tree Editor

“…Elbaum et al provide good overviews of existing prioritization approaches [26]. There has been some work on test case prioritization that considered limited resources [27], [28].…”