Test set size minimization and fault detection effectiveness: A case study in a space application

Wong, W. Eric; Horgan, J.R.; Mathur, Aditya P.; Pasquini, Alberto

doi:10.1016/s0164-1212(99)00048-5

Cited by 77 publications

(28 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In all cases there was at least an 80% average reduction in the size of the test-suite. This reduction reinforces the findings in Wong et al (1997Wong et al ( , 1998, Rothermel et al (1998), Jones and Harrold (2003) and is to be expected since our test-case generation method produces one test-case per construct of interest (variable value, transition, decision value, or MC/DC value). This generation approach is desirable since it makes traceability of the test-cases to their test-objective straightforward (one-to-one mapping).…”

Section: Test-suite Reductionsupporting

confidence: 85%

“…The effect on the fault finding capability of the reduced test-suites is, however, unclear and the studies show conflicting evidence. Wong et al (1997Wong et al ( , 1998 found no significant effect in fault finding ability between the full suites and the reduced suites. On the other hand, Rothermel et al (1998) and Jones and Harrold (2003) showed that the reduced test-suites can be dramatically worse with respect to fault finding.…”

mentioning

confidence: 90%

“…In an early study, Wong et al address the question of the effect on fault detection of reducing the size of a test set while holding coverage constant (Wong et al, 1997(Wong et al, , 1998. Their experiments were carried out over a set of commonly used UNIX utilities implemented in C. These programs were manually seeded with faults, producing variant programs each of which contained a single fault.…”

Section: Previous Test-reduction Experimentsmentioning

confidence: 99%

“…For example, a reduced test-suite T R may provide the same structural coverage as a test-suite T with significantly fewer test-cases. Previous studies conducted on C code have shown that test-suite reduction techniques significantly reduce the number of test-cases in a test-suite while maintaining the structural coverage of the original suite (Wong et al, 1997(Wong et al, , 1998Rothermel et al, 1998;Jones and Harrold, 2003). The effect on the fault finding capability of the reduced test-suites is, however, unclear and the studies show conflicting evidence.…”

mentioning

confidence: 99%

See 3 more Smart Citations

On the effect of test-suite reduction on automatically generated model-based tests

Heimdahl

Devaraj

2007

Autom Softw Eng

View full text Add to dashboard Cite

Model checking techniques can be successfully employed as a test-case generation technique to generate tests from formal models. The number of tests-cases produced, however, is typically large for complex coverage criteria such as MC/DC. Test-suite reduction can provide us with a smaller set of test-cases that preserve the original coverage-often a dramatically smaller set. Nevertheless, one potential drawback with test-suite reduction is that this might affect the quality of the test-suite in terms of fault finding. Previous empirical studies provide conflicting evidence on this issue. To further investigate the problem and determine its effect when testing implementations derived from formal models of software we performed an experiment using a large case example of a Flight Guidance System, generated reduced test-suites for a variety of structural coverage criteria while preserving coverage, and recorded their fault finding effectiveness. Our results indicate that the size of the specification based test-suites can be dramatically reduced and that the fault detection of the reduced test-suites is adversely affected. In this report we describe our experiment, analyze the results, and discuss the implications for testing based on formal specifications.

show abstract

Section: Test-suite Reductionsupporting

confidence: 85%

mentioning

confidence: 90%

Section: Previous Test-reduction Experimentsmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

On the effect of test-suite reduction on automatically generated model-based tests

Heimdahl

Devaraj

2007

Autom Softw Eng

View full text Add to dashboard Cite

show abstract

“…In Rothermel et al (1998), experiments indicate that test suite reduction may compromise fault detection. But Wong in Wong et al (1999) uses different case studies and does not notice significant fault detection loss in reduced test suites. More recently, Heimdahl stated significant fault-finding loss in other experiments (Heimdahl and Devaraj 2007).…”

Section: Introductionmentioning

confidence: 99%

Prioritizing test cases with string distances

et al. 2011

View full text Add to dashboard Cite

Test case prioritisation aims at finding an ordering which enhances a certain property of an ordered test suite. Traditional techniques rely on the availability of code or a specification of the program under test. We propose to use string distances on the text of test cases for their comparison and elaborate a prioritisation algorithm. Such a prioritisation does not require code or a specification and can be useful for initial testing and in cases when code is difficult to instrument. In this paper, we also report on experiments performed on the "Siemens Test Suite", where the proposed prioritisation technique was compared with random permutations and four classical string distance metrics were evaluated. The obtained results, confirmed by a statistical analysis, indicate that prioritisation based on string distances is more efficient in finding defects than random ordering of the test suite: the test suites prioritized using string distances are more efficient in detecting the strongest mutants, and, on average, have a better APFD than randomly ordered test suites. The results suggest that string distances can be used for prioritisation purposes, and Manhattan distance could be the best choice.

show abstract

On the suitability of checked coverage and genetic parameter tuning in test suite reduction

Koitz‐Hristov,

Sterner,

Stracke

et al. 2024

J Software Evolu Process

View full text Add to dashboard Cite

As software projects evolve and grow in size and complexity, so do their test suites. Test suite reduction (TSR) aims at reducing the size of a test suite by removing redundant and obsolete test cases based on a coverage metric while preserving its fault detection capabilities. The contributions of this paper are twofold: (1) We examine a lesser‐known coverage criterion, that is, checked coverage. Checked coverage not only investigates if a part of the code was executed but also if it was checked by a test oracle. In an empirical evaluation, we performed TSR based on different reduction algorithms, coverage metrics, and open‐source Java projects with our own TSR tool to determine the most effective and efficient combination of metric and method. (2) Given the results of the first evaluation, we further investigate the potential of parameter optimization in regard to a genetic reduction algorithm. In particular, we focus on finding a general setting for the parameters crossover rate and mutation rate such that test suites can be reduced in a reasonable time while maintaining a high fault detection power.

show abstract

Test set size minimization and fault detection effectiveness: A case study in a space application

Cited by 77 publications

References 13 publications

On the effect of test-suite reduction on automatically generated model-based tests

On the effect of test-suite reduction on automatically generated model-based tests

Prioritizing test cases with string distances

On the suitability of checked coverage and genetic parameter tuning in test suite reduction

Contact Info

Product

Resources

About