A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases

León, David De; Podgurski, Andy

doi:10.1109/issre.2003.1251065

Cited by 126 publications

(117 citation statements)

References 17 publications

Supporting

Mentioning

112

Contrasting

Unclassified

Order By: Relevance

“…The main purpose of traditional software testing is to achieve the maximum code coverage [8] and the main idea is that the test suite is capable of exercising the whole program under test. In other words, there must be some test cases per program element (including statement, branch, def-use pair and so on) which can cover it.…”

Section: Coverage-based Techniquesmentioning

confidence: 99%

“…A major difficulty with the regression testing is the excessive number of test cases accumulated while generating new test cases to test any new or modified functionality within the program [1]. To resolve the difficulty and to reduce the excessive cost of regression testing, various coverage-based and distribution-based techniques have been proposed [2,3,4,5,6,7,8]. These techniques attempt to permanently discard redundant test cases and retain the most effective ones to reduce the excessive cost of regression testing [6].…”

Section: Introductionmentioning

confidence: 99%

“…A number of recent papers have investigated techniques for test suite reduction using execution profiles, applied in distribution-based techniques [8]. An execution profile generated by a test case identifies the program elements covered by the test case.…”

Section: Introductionmentioning

confidence: 99%

“…Observation-based testing is potentially applicable in situations where it is desirable to filter a large amount of test cases identifying a promising subset of test cases for conformance to the requirements [13]. Despite the effectiveness in detecting failures, observation based methods do not necessarily provide full coverage [8]. Moreover, empirical studies [8] show that the functionality of these methods varies with the size of the test suite.…”

Section: Introductionmentioning

confidence: 99%

“…Despite the effectiveness in detecting failures, observation based methods do not necessarily provide full coverage [8]. Moreover, empirical studies [8] show that the functionality of these methods varies with the size of the test suite. These studies also indicate that coveragebased and distribution-based techniques often do not select similar test cases and are complementary because they find different faults.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Bi-criteria Test Suite Reduction by Cluster Analysis of Execution Profiles

Khalilian

Parsa

2012

Advances in Software Engineering Techniques

View full text Add to dashboard Cite

Abstract. The aim has been to minimize regression test suites while retaining fault detection capability of the test suite admissible. An appropriate minimized test suite should exercise different execution paths within a program. However, minimization of test suites may result in significant fault detection loss. To alleviate the loss, a new bi-criteria heuristic algorithm, using cluster analysis of test cases execution profiles is proposed in this paper. Cluster analysis of execution profiles categorizes test cases according to their similarity in terms of exercising a certain coverage criterion. Considering additional coverage criteria the proposed algorithm samples some test cases from each cluster. These additional criteria exercise execution paths, different from those covered by the main testing criteria. Experiments on the Siemens suite manifest the applicability of the proposed approach and present interesting insights into the use of cluster analysis to the bi-criteria test suite reduction.

show abstract

Section: Coverage-based Techniquesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Bi-criteria Test Suite Reduction by Cluster Analysis of Execution Profiles

Khalilian

Parsa

2012

Advances in Software Engineering Techniques

View full text Add to dashboard Cite

show abstract

Regression testing minimization, selection and prioritization: a survey

Yoo¹,

Harman²

2012

Softw. Test. Verif. Reliab.

1,009

592

View full text Add to dashboard Cite

Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research.This survey aims to collect and consider papers that deal with three regression testing techniques: test suite minimization, RTS and test case prioritization. Our intention is not to undertake a systematic review, but rather to provide a broad state-of-the-art view on these related fields. Many different approaches have been proposed to aid regression testing, which has resulted in a body of literature that is spread over a wide variety of domains and publication venues. The majority of surveyed literature has been published in the software engineering domain, and especially in the software testing and software maintenance literature. However, the regression testing literature also overlaps with those of programming language analysis, empirical software engineering and software metrics.Therefore, the paper selection criteria on which this survey is based are the problems considered in papers, while focusing on the specific topics of minimization, selection and prioritization. The formal definitions of these problems are presented in Section 2.2. The selected papers are listed in the Appendix. Fast abstracts and short papers have been excluded. BACKGROUNDThis section introduces the basic concepts and definitions that form a nomenclature of regression testing and minimization, selection and prioritization techniques. Definition 1 (Test Suite Minimization Problem)Given: A test suite, T , a set of test requirements {r 1 , . . .,r n }, that must be satisfied to provide the desired 'adequate' testing of the program, and subsets of T , T 1 , . . ., T n , one associated with each of the r i s such that any one of the test cases t j belonging to T i can be used to achieve requirement r i .Problem: Find a representative set, T , of test cases from T that satisfies all r i s.The testing criterion is satisfied when every test requirement in {r 1 , . . .,r n } is satisfied. A test requirement, r i , is satisfied by any test case, t j , that belongs to the T i , a subset of T . Therefore, the representative set of test cases is the hitting set of the T i s. Furthermore, in order to maximize the effect of minimization, T should be the minimal hitting set of the T i ...

show abstract

Prioritizing manual test cases in rapid release environments

Hemmati

Fang

Mäntylä

et al. 2016

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Test case prioritization is an important testing activity, in practice, specially for large scale systems. The goal is to rank the existing test cases in a way that they detect faults as soon as possible, so that any partial execution of the test suite detects the maximum number of defects for the given budget. Test prioritization becomes even more important when the test execution is time consuming, for example, manual system tests versus automated unit tests. Most existing test case prioritization techniques are based on code coverage, which requires access to source code. However, manual testing is mainly performed in a black‐box manner (manual testers do not have access to the source code). Therefore, in this paper, the existing test case prioritization techniques (e.g. diversity‐based and history‐based techniques) are examined and modified to be applicable on manual black‐box system testing. An empirical study on four older releases of desktop Firefox showed that none of the techniques were strongly dominating the others in all releases. However, when nine more recent releases of desktop Firefox, where the development has been moved from a traditional to a more agile and rapid release environment, were studied, a very significant difference between the history‐based approach and its alternatives was observed. The higher effectiveness of the history‐based approach compared with alternatives also held on 28 additional rapid releases of other Firefox projects – mobile Firefox and tablet Firefox. The conclusion of the paper is that test cases in rapid release environments can be very effectively prioritized for execution, based on their historical failure knowledge. In particular, it is the recency of historical knowledge that explains its effectiveness in rapid release environments rather than other changes in the process. Copyright © 2016 John Wiley & Sons, Ltd.

show abstract

A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases

Cited by 126 publications

References 17 publications

Bi-criteria Test Suite Reduction by Cluster Analysis of Execution Profiles

Bi-criteria Test Suite Reduction by Cluster Analysis of Execution Profiles

Regression testing minimization, selection and prioritization: a survey

Prioritizing manual test cases in rapid release environments

Contact Info

Product

Resources

About