Test set size minimization and fault detection effectiveness: a case study in a space application

Wong, W. Eric; Horgan, J.R.; Mathur, Aditya P.; Pasquini, Alberto

doi:10.1109/cmpsac.1997.625062

Cited by 78 publications

(78 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…There are also empirical studies on the effect of test-suite reduction on lowering the fault-detection capability. However, the findings are non-conclusive: Wong et al [14], [15] found that test-suite reduction does not substantially lower the fault-detection capability of test suites, whereas Rothermel et al [16] found that test-suite reduction can severely lower the fault-detection capability.…”

Section: Introductionmentioning

confidence: 71%

“…(1) The subject programs in our study are larger than those used in previous studies; our programs are in Java not in C as in most existing reearch [2], [4], [9], [15], [17], and coding styles in Java and C can differ. (2) The nature of tests is substantially different: we focus on tests for small units of code from a larger system and hence some tests may have mutually disjoint execution paths, whereas previous studies looked at small programs where many tests execute the same main method albeit on different inputs.…”

Section: Introductionmentioning

confidence: 93%

“…(3) Test generation is substantially different: our focus is on real test suites that are accumulated during a system's evolution, whereas previous studies mainly focused on synthetic suites that were generated in the laboratory. (4) The cost-benefit analysis in our study is substantially more extensive: some previous studies considered both benefits (reduction of test suite size) and costs (lowering of fault-detection capability) of test-suite reduction [4], [14], [15] but did not consider the influence of reduction techniques and test granularities; similarly, there are studies considering the influence of reduction techniques [3], [9], [10], but they only considered benefits and not costs of test-suite reduction. In addition, no previous study evaluated the benefits/costs of different JUnit test granularities, where granulatiry refers to grouping JUnit test cases based on their test methods or their test classes.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Empirical Study of JUnit Test-Suite Reduction

Zhang

Marinov

et al. 2011

2011 IEEE 22nd International Symposium on Software Reliability Engineering

View full text Add to dashboard Cite

Abstract-As test suites grow larger during software evolution, regression testing becomes expensive. To reduce the cost of regression testing, test-suite reduction aims to select a minimal subset of the original test suite that can still satisfy all the test requirements. While traditional test-suite reduction techniques were intensively studied on C programs with specially generated test suites, there are limited studies for test-suite reduction on programs with real-world test suites. In this paper, we investigate test-suite reduction techniques on Java programs with real-world JUnit test suites. We implemented four representative test-suite reduction techniques for JUnit test suites. We performed an empirical study on 19 versions of four real-world Java programs, ranging from 1.89 KLoC to 80.44 KLoC. Our study investigates both the benefits and the costs of test-suite reduction. The results show that the four traditional test-suite reduction techniques can effectively reduce these JUnit test suites without substantially reducing their fault-detection capability. Based on the results, we provide a guideline for achieving cost-effective JUnit testsuite reduction.

show abstract

Section: Introductionmentioning

confidence: 71%

Section: Introductionmentioning

confidence: 93%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Empirical Study of JUnit Test-Suite Reduction

Zhang

Marinov

et al. 2011

2011 IEEE 22nd International Symposium on Software Reliability Engineering

View full text Add to dashboard Cite

show abstract

“…Another research question of relevance is whether minimization of the test-set size has any impact on fault detection effectiveness [35]. It was found that the minimization of test-sets had minimal effect.…”

Section: Related Workmentioning

confidence: 99%

Experimental evaluation of the tolerance for control‐flow test criteria

Kapoor

Bowen

2004

Software Testing Verif & Rel

View full text Add to dashboard Cite

For a given test criterion, the number of test-sets satisfying the criterion may be very large, with varying fault detection effectiveness. In recent work [29], the measure of variation in effectiveness of test criterion was defined as 'tolerance'. This paper presents an experimental evaluation of tolerance for control-flow test criteria. The experimental analysis is done by exhaustive test-set generation, wherever possible, for a given criteria which improves on earlier empirical studies that adopted analysis of some test-sets using random selection techniques. Four industrially used control-flow testing criteria, Condition Coverage (CC), Decision Condition Coverage (DCC), Full Predicate Coverage (FPC) and Modified Condition/Decision Coverage (MC/DC) have been analysed against four types of faults. A new test criteria, Reinforced Condition/Decision Coverage (RC/DC) [28], is also analysed and compared. The Boolean specifications considered were taken from a past research paper and also generated randomly. To ensure that it is the test-set property that influences the effectiveness and not the test-set size, the average test-set size was kept same for all the test criteria except RC/DC. A further analysis of variation in average effectiveness with respect to number of conditions in the decision was also undertaken. The empirical results show that the MC/DC criterion is more reliable and stable when compared to the other considered criteria. Though the number of test-cases is large in RC/DC testsets, no significant improvement in effectiveness and tolerance was observed over MC/DC.

show abstract

“…It also enlarges the size of a test suite unnecessarily. Moreover, the effectiveness of the test suites may be affected if test suite minimization techniques [106,105,30] are used because some legitimate test case will be removed during test suite minimization. Thirdly, an abstraction through reduction should include all traces and all failures of the specification.…”

Section: Justification Of Our Proposalmentioning

confidence: 99%

Towards a new extension relation for compositional test case generation for CSP concurrent processes

Chan¹,

陳榮光.²

View full text Add to dashboard Cite

Software concurrent systems such as electronic financial services are very popular today. Rigorous testing of such systems is indispensable. Nevertheless, the state-explosion problem, in which the size of a system grows exponentially with the number of concurrent components, is a severe obstacle to the generation of test cases.In this study, we propose a new relationship assuring the conformities of test cases for concurrent systems. We stipulate it in Communicating Sequential Processes (CSP), which is an excellent tool to study concurrent systems. Specification of concurrent systems is expressed as processes that are composed sequentially and concurrently. Our proposal supports both abstraction and process composition.This approach decomposes a given process into sequential compositions of component processes. Each component can be anti-extended into abstract forms. Sequential and concurrent combinations of abstract forms can substitute their corresponding extending components in a process to form aggregated abstract forms. Since the given processes, components and abstract forms are all processes, this approach can be applied recursively and compositionally. We prove that these aggregated abstract processes should be anti-extensions of the corresponding processes under a few necessary and sufficient conditions. Hence, test cases generated from these processes will be conformance test cases for the implementations.We examine three major testing relations (conformance, extension and reduction) from the point of view of their ability to support abstraction. Extension and reduction are special kinds of conformance, which are partially ordered. However, only extension from a specification can provide abstract forms that guarantee to be conformed by an implementation of the specification. Extension is further investigated to support the compositional approach. We identify five weaknesses of extension, which we call problems of preceding, succeeding, prefix, decomposition and paralleling. Each will result in non-conforming abstraction.To resolve the identified weaknesses of extension in a compositional and abstractionoriented approach, we formulate σ-extension and identify the necessary and sufficient conditions. We also discuss a restricted sub-class of process specification so that these conditions can easily be satisfied. In the restricted sub-class, successful terminations of processes are deterministic.Under these conditions, we achieve a few desirable properties. First, both preceding and succeeding components of a given sequential composition can be anti-extended. The given composition should extend the sequential composition of its corresponding anti-extended processes. Secondly, a given parallel composition whose concurrent components are sequential compositions should extend a parallel composition of anti-extensions of preceding components of corresponding sequential compositions, followed by a parallel composition of anti-extensions of corresponding succeeding components. Furthermore, a given sequential co...

show abstract

Test set size minimization and fault detection effectiveness: a case study in a space application

Cited by 78 publications

References 10 publications

An Empirical Study of JUnit Test-Suite Reduction

An Empirical Study of JUnit Test-Suite Reduction

Experimental evaluation of the tolerance for control‐flow test criteria

Towards a new extension relation for compositional test case generation for CSP concurrent processes

Contact Info

Product

Resources

About