Effectively prioritizing tests in development environment

Srivastava, Amitabh; Thiagarajan, Jay

doi:10.1145/566171.566187

Cited by 98 publications

(48 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Srivastava and Thiagarajan combined the greedy-based prioritization approach with RTS [126]. They first identified the modified code blocks in the new version of the SUT by comparing its binary code to that of the previous version.…”

Section: Coverage-based Prioritizationmentioning

confidence: 99%

Regression testing minimization, selection and prioritization: a survey

Yoo¹,

Harman²

2012

Softw. Test. Verif. Reliab.

1,041

594

View full text Add to dashboard Cite

Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research.This survey aims to collect and consider papers that deal with three regression testing techniques: test suite minimization, RTS and test case prioritization. Our intention is not to undertake a systematic review, but rather to provide a broad state-of-the-art view on these related fields. Many different approaches have been proposed to aid regression testing, which has resulted in a body of literature that is spread over a wide variety of domains and publication venues. The majority of surveyed literature has been published in the software engineering domain, and especially in the software testing and software maintenance literature. However, the regression testing literature also overlaps with those of programming language analysis, empirical software engineering and software metrics.Therefore, the paper selection criteria on which this survey is based are the problems considered in papers, while focusing on the specific topics of minimization, selection and prioritization. The formal definitions of these problems are presented in Section 2.2. The selected papers are listed in the Appendix. Fast abstracts and short papers have been excluded. BACKGROUNDThis section introduces the basic concepts and definitions that form a nomenclature of regression testing and minimization, selection and prioritization techniques. Definition 1 (Test Suite Minimization Problem)Given: A test suite, T , a set of test requirements {r 1 , . . .,r n }, that must be satisfied to provide the desired 'adequate' testing of the program, and subsets of T , T 1 , . . ., T n , one associated with each of the r i s such that any one of the test cases t j belonging to T i can be used to achieve requirement r i .Problem: Find a representative set, T , of test cases from T that satisfies all r i s.The testing criterion is satisfied when every test requirement in {r 1 , . . .,r n } is satisfied. A test requirement, r i , is satisfied by any test case, t j , that belongs to the T i , a subset of T . Therefore, the representative set of test cases is the hitting set of the T i s. Furthermore, in order to maximize the effect of minimization, T should be the minimal hitting set of the T i ...

show abstract

Section: Coverage-based Prioritizationmentioning

confidence: 99%

Regression testing minimization, selection and prioritization: a survey

Yoo¹,

Harman²

2012

Softw. Test. Verif. Reliab.

1,041

594

View full text Add to dashboard Cite

show abstract

“…Regression test prioritization techniques reorder test executions in order to maximize a certain objective function, such as exposing faults earlier or reducing the execution time cost [37]. For insatnce, Microsoft researchers have built test prioritization systems for development and maintenance of Windows for a decade [48,10]. Moreover, a large body of research work has been proposed to design and evaluate regression TCP techniques [54,59,45,44,37,33].…”

Section: Introductionmentioning

confidence: 99%

A large-scale empirical comparison of static and dynamic test case prioritization techniques

Luo

Moran

Poshyvanyk

2016

Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

View full text Add to dashboard Cite

The large body of existing research in Test Case Prioritization (TCP) techniques, can be broadly classified into two categories: dynamic techniques (that rely on run-time execution information) and static techniques (that operate directly on source and test code). Absent from this current body of work is a comprehensive study aimed at understanding and evaluating the static approaches and comparing them to dynamic approaches on a large set of projects.In this work, we perform the first extensive study aimed at empirically evaluating four static TCP techniques comparing them with state-of-research dynamic TCP techniques at different test-case granularities (e.g., method and classlevel) in terms of effectiveness, efficiency and similarity of faults detected. This study was performed on 30 real-word Java programs encompassing 431 KLoC. In terms of effectiveness, we find that the static call-graph-based technique outperforms the other static techniques at test-class level, but the topic-model-based technique performs better at testmethod level. In terms of efficiency, the static call-graphbased technique is also the most efficient when compared to other static techniques. When examining the similarity of faults detected for the four static techniques compared to the four dynamic ones, we find that on average, the faults uncovered by these two groups of techniques are quite dissimilar, with the top 10% of test cases agreeing on only ≈ 25% -30% of detected faults. This prompts further research into the severity/importance of faults uncovered by these techniques, and into the potential for combining static and dynamic information for more effective approaches.

show abstract

“…Various techniques for test case prioritization have been proposed in research literature by several researchers. These techniques have addressed test case prioritization according to rate of fault detection or code coverage capabilities and are evaluated through various empirical studies [3,4,5,6,7,8,9]. Rummel, Kapfhammer, and Thall [10] suggest that test suite can be prioritized according to all DU's with minimal time and space overhead.…”

Section: Background and Related Workmentioning

confidence: 99%

“…There are many search techniques for test case prioritization, which are being developed and unfolded by various researchers in the field [13]. Further, in recent past many techniques and studies have been proposed [4,6,7,9,14,15,16,17,18,19] for the prioritization of test cases for regression testing.…”

Section: Background and Related Workmentioning

confidence: 99%

Empirical Validation of Variable based Test Case Prioritization/Selection Technique

Singh¹

2009

JDCTA

View full text Add to dashboard Cite

Regression test case prioritization optimizes the ordering of test cases to be executed to meet some criteria like maximum code coverage or high rate of fault detection. In prior work we prioritized test cases according to a hybrid technique using variable based method that combined both selection and prioritization. We inferred in our approach that variables are vital source of changes in the program and test cases should be prioritized according to the variables of any changed statement and variables computed from the variables of changed statements. In support of our prioritization approach we extend our work to validate the effectiveness of prioritized test cases with respect to data flow technique. This paper reports an experimental study investigating the effectiveness of our prioritization approach given in [2] by considering programs. The results obtained are encouraging and support our work to validate the prioritization technique with respect to DU/DC paths of data flow testing technique.analysis of the results are presented in section 4 and conclusions are given in Section 5.

show abstract

Effectively prioritizing tests in development environment

Cited by 98 publications

References 22 publications

Regression testing minimization, selection and prioritization: a survey

Regression testing minimization, selection and prioritization: a survey

A large-scale empirical comparison of static and dynamic test case prioritization techniques

Empirical Validation of Variable based Test Case Prioritization/Selection Technique

Contact Info

Product

Resources

About