Java Unit Testing Tool Competition - Seventh Round

Abstract

We report on the results of the seventh edition of the JUnit tool competition. This year, four tools were executed on a benchmark with (i) new classes, selected from real-world software projects, and (ii) challenging classes from the previous edition. We use Randoop and manual test suites from the projects as baselines. Given the interesting findings of last year, we analyzed the effectiveness of the combined test suites generated by all competing tools and compared; results are confronted with the manual test suites of the projects, as well as those generated by the competing tools . This paper describes our methodology and the results, highlight challenges faced during the contest.

Java unit testing tool competition

“…• PdfBox (https://github.com/apache/pdfbox), an Apache Java API to work with PDF documents. • Spoon (https://github.com/INRIA/spoon/), a library for analyzing and transforming Java source code, also used in last year's edition of the competition [6].…”

“…Considering all the classes in each project is not possible as it would require an extensive amount of time and resources for the competition. Following the lead of past editions [6], we sampled a limited number of Classes under test (CUTs) using a two-step procedure. In the first step, we computed McCabe's cyclomatic complexity for all methods and classes in each project using Ja-vaNCSS 1 and filtered out classes that contain only methods with a complexity lower than five.…”

“…Additionally, the infrastructure includes statistical analysis scripts to compare and rank the different test case generation tools. This year, we implemented several improvements to our infrastructure, including parallelizing the execution of mutation analysis, fixing the issues reported during last year's edition of the competition [6], and implementing a novel Docker image to execute the Java unit test case generators. The infrastructure is open-source and available on GitHub at https://github.com/JUnitContest/junitcontest.…”

Ant Colony Optimization for Object-Oriented Unit Test Generation

“…Testing (SBST) tool competition, where it achieved the second highest score out of five tools in 2019 [19]. EvoSuite is the state of the art in search-based unit test generation [13].…”

Second Competition on Software Testing: Test-Comp 2020

“…There are two competitions for automatic and off-site testing: Rode0day 8 is a competition that is meant as a continuously running evaluation on bug-finding in binaries (currently Grep and SQLite). The unit-testing tool competition [32] 9 is part of the SBST workshop and compares tools for unit-test generation on Java programs. There was no comparative evaluation of automatic test-generation tools for whole C programs in source-code, in a controlled environment, and Test-Comp was founded to close this gap [4].…”