Unit Testing Tool Competition -- Round Three

Rueda, U.; Vos, Tanja E. J.; Prasetya, I. S. W. B.

doi:10.1109/sbst.2015.12

Cited by 23 publications

(56 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The context of our study is a random selection of classes from four test benchmarks: (i) the SF110 corpus [18]; (ii) the SBST tool contest 2015 [53]; (iii) the SBST tool contest 2016 [49]; and (iv) the benchmark used in our previous conference paper [42].…”

Section: Subjectsmentioning

confidence: 99%

“…From the latest two editions of the SBST unit test tool contests [53], [49] we selected 97 subjects: 46 classes from the third edition [53] and 51 classes from the fourth edition [49] of the contest. From the original set of classes in the two competitions, we removed all duplicate classes belonging to different versions of the same library since we would not expect any relevant difference over different versions in terms of branch, statement, or mutation coverage.…”

Section: Subjectsmentioning

confidence: 99%

“…Second, we conduct a large empirical study involving 346 Java classes sampled from four different datasets [18], [53], [49], [42] and considering three well known test adequacy criteria: statement coverage, branch coverage and mutation coverage. We find that DynaMOSA achieves significantly higher coverage than WSA in a large number of classes (27% and 28% for branch and mutation coverage respectively) with an average increment, for classes where statistically significant differences were observed, of +8% for branch coverage and +11% for mutation coverage.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

Panichella

Kifetew

Tonella

2018

IIEEE Trans. Software Eng.

247

290

View full text Add to dashboard Cite

Abstract-The test case generation is intrinsically a multi-objective problem, since the goal is covering multiple test targets (e.g., branches). Existing search-based approaches either consider one target at a time or aggregate all targets into a single fitness function (whole-suite approach). Multi and many-objective optimisation algorithms (MOAs) have never been applied to this problem, because existing algorithms do not scale to the number of coverage objectives that are typically found in real-world software. In addition, the final goal for MOAs is to find alternative trade-off solutions in the objective space, while in test generation the interesting solutions are only those test cases covering one or more uncovered targets. In this paper, we present DynaMOSA (Dynamic Many-Objective Sorting Algorithm), a novel many-objective solver specifically designed to address the test case generation problem in the context of coverage testing. DynaMOSA extends our previous many-objective technique MOSA (Many-Objective Sorting Algorithm) with dynamic selection of the coverage targets based on the control dependency hierarchy. Such extension makes the approach more effective and efficient in case of limited search budget. We carried out an empirical study on 346 Java classes using three coverage criteria (i.e., statement, branch, and strong mutation coverage) to assess the performance of DynaMOSA with respect to the whole-suite approach (WS), its archive-based variant (WSA) and MOSA. The results show that DynaMOSA outperforms WSA in 28% of the classes for branch coverage (+8% more coverage on average) and in 27% of the classes for mutation coverage (+11% more killed mutants on average). It outperforms WS in 51% of the classes for statement coverage, leading to +11% more coverage on average. Moreover, DynaMOSA outperforms its predecessor MOSA for all the three coverage criteria in 19% of the classes with +8% more code coverage on average.

show abstract

Section: Subjectsmentioning

confidence: 99%

Section: Subjectsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

Panichella

Kifetew

Tonella

2018

IIEEE Trans. Software Eng.

247

290

View full text Add to dashboard Cite

show abstract

“…Finally, the column tuned shows the coverage delivered by tuned T3i. Table 4 shows the hypothetical strength of the queries we wrote with T3i (only Hoare triple queries are used in this experiment), measured in terms of their ability to detect mutations (bugs), artificially injected using the SBST benchmarking tool [9]. The column qkill shows the percentage of the mutations found (killed) by the queries.…”

Section: Cost and Benefitmentioning

confidence: 99%

“…A recent tool contest [9] shows that even the best scoring tool can only deliver 50% branch coverage over all target classes (63) in the benchmark. The contest was setup such that the target classes are not known upfront.…”

Section: Introductionmentioning

confidence: 99%

T3i: a tool for generating and querying test suites for Java

Prasetya

2015

Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

View full text Add to dashboard Cite

T3i is an automated unit-testing tool to test Java classes. To expose interactions T3i generates test-cases in the form of sequences of calls to the methods of the target class. What separates it from other testing tools is that it treats test suites as first class objects and allows users to e.g. combine, query, and filter them. With these operations, the user can construct a test suite with specific properties. Queries can be used to check correctness properties. Hoare triples, LTL formulas, and algebraic equations can be queried. T3i can be used interactively, thus facilitating more exploratory testing, as well as through a script. The familiar Java syntax can be used to control it, or alternatively one can use the much lighter Groovy syntax.

show abstract

Evaluating code‐based test input generator tools

Cseppento

Micskei

2017

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary In recent years, several tools have been developed to automatically select test inputs from the code of the system under test. However, each of these tools has different advantages, and there is a little detailed feedback available on the actual capabilities of the various tools. To evaluate test input generators, this paper collects a set of programming language concepts that should be handled by the tools and maps these core concepts and challenging features like handling the environment or multi‐threading to 363 code snippets, respectively. These snippets would serve as inputs for the tools. Next, the paper presents SETTE, an automated framework to execute and evaluate these snippets. Using SETTE, multiple experiments were performed on five Java and one .NET‐based tools using symbolic execution, search‐based, and random techniques. The test suites' coverage, size, generation time, and mutation score were compared. The results highlight the strengths and weaknesses of each tool and approach and identify hard code parts that are difficult to tackle for most of the tools. We hope that this research could serve as actionable feedback to tool developers and help practitioners assess the readiness of test input generation.

show abstract

Unit Testing Tool Competition -- Round Three

Cited by 23 publications

References 10 publications

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

T3i: a tool for generating and querying test suites for Java

Evaluating code‐based test input generator tools

Contact Info

Product

Resources

About