A detailed investigation of the effectiveness of whole test suite generation

Rojas, José Miguel; Vivanti, Mattia; Arcuri, Andrea; Fraser, Gordon

doi:10.1007/s10664-015-9424-2

Cited by 76 publications

(72 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We selected this benchmark since it has been widely used in the literature to assess test case generation tools [18], [51], [48]. However, Shamshiri et al [51] showed that the vast majority of classes in the FS110 corpus are trivial to cover fully, even with random search algorithms.…”

Section: Subjectsmentioning

confidence: 99%

“…Recently, Rojas et al [48] developed the Whole Suite with Archive approach (WSA), a hybrid strategy that incorporates some of MOSA's routines inside the traditional WS approach. While WSA still applies the sum scalarization and works at the test suite level, it incorporates an archive strategy which operates at the test case level.…”

Section: Introductionmentioning

confidence: 99%

“…From an empirical point of view, WSA has been proved to be statistically superior to WS and to the one-target at a time approaches. However, from a theoretical point of view, it does not evolve test suites anymore since the final test suite given to developers is artificially synthesised by taking those test cases stored in the archive rather than returning the best individual (i.e., test suite) from the last generation of GAs [48]. Rojas et al [48] arose the following still unanswered questions: (1) To what extent do the benefits of MOSA stem from the many-objective reformulation of the problem or from the use of the archiving mechanism?…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

Panichella

Kifetew

Tonella

2018

IIEEE Trans. Software Eng.

257

300

View full text Add to dashboard Cite

Abstract-The test case generation is intrinsically a multi-objective problem, since the goal is covering multiple test targets (e.g., branches). Existing search-based approaches either consider one target at a time or aggregate all targets into a single fitness function (whole-suite approach). Multi and many-objective optimisation algorithms (MOAs) have never been applied to this problem, because existing algorithms do not scale to the number of coverage objectives that are typically found in real-world software. In addition, the final goal for MOAs is to find alternative trade-off solutions in the objective space, while in test generation the interesting solutions are only those test cases covering one or more uncovered targets. In this paper, we present DynaMOSA (Dynamic Many-Objective Sorting Algorithm), a novel many-objective solver specifically designed to address the test case generation problem in the context of coverage testing. DynaMOSA extends our previous many-objective technique MOSA (Many-Objective Sorting Algorithm) with dynamic selection of the coverage targets based on the control dependency hierarchy. Such extension makes the approach more effective and efficient in case of limited search budget. We carried out an empirical study on 346 Java classes using three coverage criteria (i.e., statement, branch, and strong mutation coverage) to assess the performance of DynaMOSA with respect to the whole-suite approach (WS), its archive-based variant (WSA) and MOSA. The results show that DynaMOSA outperforms WSA in 28% of the classes for branch coverage (+8% more coverage on average) and in 27% of the classes for mutation coverage (+11% more killed mutants on average). It outperforms WS in 51% of the classes for statement coverage, leading to +11% more coverage on average. Moreover, DynaMOSA outperforms its predecessor MOSA for all the three coverage criteria in 19% of the classes with +8% more code coverage on average.

show abstract

Section: Subjectsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

Panichella

Kifetew

Tonella

2018

IIEEE Trans. Software Eng.

257

300

View full text Add to dashboard Cite

show abstract

“…Whole test suite generation. Whole test suite generation builds an entire test suite by simultaneously optimizing multiple fitness functions (e.g., for multiple coverage criteria) [34], [35]. In principle and with appropriate fitness functions defined for validity and representativeness, the problem addressed in this paper can be formulated as whole test suite generation.…”

Section: Related Workmentioning

confidence: 99%

Synthetic data generation for statistical testing

Soltana

Sabetzadeh

Briand

2017

2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)

View full text Add to dashboard Cite

Abstract-Usage-based statistical testing employs knowledge about the actual or anticipated usage profile of the system under test for estimating system reliability. For many systems, usagebased statistical testing involves generating synthetic test data. Such data must possess the same statistical characteristics as the actual data that the system will process during operation. Synthetic test data must further satisfy any logical validity constraints that the actual data is subject to. Targeting data-intensive systems, we propose an approach for generating synthetic test data that is both statistically representative and logically valid. The approach works by first generating a data sample that meets the desired statistical characteristics, without taking into account the logical constraints. Subsequently, the approach tweaks the generated sample to fix any logical constraint violations. The tweaking process is iterative and continuously guided toward achieving the desired statistical characteristics. We report on a realistic evaluation of the approach, where we generate a synthetic population of citizens' records for testing a public administration IT system. Results suggest that our approach is scalable and capable of simultaneously fulfilling the statistical representativeness and logical validity requirements.

show abstract

“…The more coverage they TS will achieve the better chance will the contracts have of capturing errors. It is therefore proposed to use Whole Test Suite Generation instead of proposed to use Whole Test Suite (WTS) [5] Generation instead of randomly generation to create the initial TS for AutoInfer. WTS is the state of the art that uses the genetic Algorithm (GA) to generate the entire TS evolutionally at the same time.…”

Section: Introductionmentioning

confidence: 99%

Invariant Detection Using Enhanced Autoinfer

Ashraf¹,

Abbasi²

2017

JSW

View full text Add to dashboard Cite

Abstract:AutoInfer is a tool that is the state of the art in invariant detection. Invariants are properties of program components that remain unchanged throughout the execution of that component. AutoInfer automatically detects invariants for the programs under test which may or may not have a few invariant already present in them. AutoInfer uses AutoTest to generate a test suite for a given Program under test. The test suite (TS) is generated randomly using routine coverage as the coverage criteria. The generated TS is run on the program to create a change profile for the program, which in turn is used to activate relevant templates to generate quantified expressions. These expressions are the candidate contracts. Candidate contracts when run against the test cases are validated if they do not fail any test case. AutoInfer's results are heavily based on the generated Test Suite. The better the generated Test Suite is better the final contracts will be. The work in this paper proposes that using Whole Test Suite (WTS) generation instead of random generation to generate the TS will results in a faster generation of the TS and will capture more errors as compared to AutoInfer. WTS is the state of the art in TS generation so it will result in better coverage and an improved fault detection capability.

show abstract

A detailed investigation of the effectiveness of whole test suite generation

Cited by 76 publications

References 28 publications

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

Synthetic data generation for statistical testing

Invariant Detection Using Enhanced Autoinfer

Contact Info

Product

Resources

About