GRT: Program-Analysis-Guided Random Testing (T)

Ма, Лей; Artho, Cyrille; Zhang, Cheng; Satō, Hiroyuki; Gmeiner, Johannes; Ramler, Rudolf

doi:10.1109/ase.2015.49

Cited by 56 publications

(55 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unconfirmed cases are counted as unknown. Finally, 23 issues were confirmed by developers, and most of our reported defects have already been fixed [9]. …”

Section: B Evaluation Of Generated Testssupporting

confidence: 55%

“…GRT [9] is a fully automated test generator that uses knowledge extracted from software under test (SUT) to guide each step of test generation to achieve high code coverage and bug detection ability [10]. GRT uses random test generator Randoop [11] as its basic framework, which generates tests iteratively.…”

Section: A Grtmentioning

confidence: 99%

“…They relate to 56 distinct issues in the given software, out of which 23 test cases pertained to confirmed defects [9]. We analyze all issues and categorize them based on the failure type, whether they were true or false positives, and how they were fixed.…”

Section: Introductionmentioning

confidence: 99%

“…We applied GRT, a random test case generator, to the most recent version of real-world popular software projects, with the goal of uncovering new, previously unknown defects [9].…”

Section: Open Source Case Studymentioning

confidence: 99%

“…Machine-readable specifications allow a tool to distinguish between expected and unexpected failures; they could become essential for reducing the false-positive rate of automatic test generation tools. This paper is organized as follows: Section II describes the case study we conducted on ten open-source real-world software libraries by using automated test generator GRT [9]. Section III discusses previous work on bug detection and classification by using automated test generators.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Classification of Randomly Generated Test Cases

Artho

Ма

2016

2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)

Self Cite

View full text Add to dashboard Cite

Abstract-Random test case generation produces relatively diverse test sequences, but the validity of the test verdict is always uncertain. Because tests are generated without taking the specification and documentation into account, many tests are invalid. To understand the prevalent types of successful and invalid tests, we present a classification of 56 issues that were derived from 208 failed, randomly generated test cases. While the existing workflow successfully eliminated more than half of the tests as irrelevant, half of the remaining failed tests are false positives. We show that the new @NonNull annotation of Java 8 has the potential to eliminate most of the false positives, highlighting the importance of machine-readable documentation.

show abstract

“…Unconfirmed cases are counted as unknown. Finally, 23 issues were confirmed by developers, and most of our reported defects have already been fixed [9]. …”

Section: B Evaluation Of Generated Testssupporting

confidence: 55%

Section: A Grtmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…We applied GRT, a random test case generator, to the most recent version of real-world popular software projects, with the goal of uncovering new, previously unknown defects [9].…”

Section: Open Source Case Studymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Classification of Randomly Generated Test Cases

Artho

Ма

2016

2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)

Self Cite

View full text Add to dashboard Cite

show abstract

JUGE: An infrastructure for benchmarking Java unit test generators

Devroey

Gambi

Galeotti

et al. 2022

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Researchers and practitioners have designed and implemented various automated test case generators to support effective software testing. Such generators exist for various languages (e.g., Java, C#, or Python) and various platforms (e.g., desktop, web, or mobile applications). The generators exhibit varying effectiveness and efficiency, depending on the testing goals they aim to satisfy (e.g., unit‐testing of libraries versus system‐testing of entire applications) and the underlying techniques they implement. In this context, practitioners need to be able to compare different generators to identify the most suited one for their requirements, while researchers seek to identify future research directions. This can be achieved by systematically executing large‐scale evaluations of different generators. However, executing such empirical evaluations is not trivial and requires substantial effort to select appropriate benchmarks, setup the evaluation infrastructure, and collect and analyse the results. In this Software Note, we present our JUnit Generation Benchmarking Infrastructure (JUGE) supporting generators (search‐based, random‐based, symbolic execution, etc.) seeking to automate the production of unit tests for various purposes (validation, regression testing, fault localization, etc.). The primary goal is to reduce the overall benchmarking effort, ease the comparison of several generators, and enhance the knowledge transfer between academia and industry by standardizing the evaluation and comparison process. Since 2013, several editions of a unit testing tool competition, co‐located with the Search‐Based Software Testing Workshop, have taken place where JUGE was used and evolved. As a result, an increasing amount of tools (over 10) from academia and industry have been evaluated on JUGE, matured over the years, and allowed the identification of future research directions. Based on the experience gained from the competitions, we discuss the expected impact of JUGE in improving the knowledge transfer on tools and approaches for test generation between academia and industry. Indeed, the JUGE infrastructure demonstrated an implementation design that is flexible enough to enable the integration of additional unit test generation tools, which is practical for developers and allows researchers to experiment with new and advanced unit testing tools and approaches.

show abstract

An Experimental Study on Flakiness and Fragility of Randoop Regression Test Suites

Paydar¹,

Azamnouri²

2019

Fundamentals of Software Engineering

View full text Add to dashboard Cite

Randoop is a well-known tool that proposes a feedbackdirected algorithm for automatic and random generation of unit tests for a given Java class. It automatically generates two test suites for the class under test: 1) an error-revealing test suite, and 2) a regression test suite. Despite successful experiences with applying Randoop on real world projects like Java Development Kit (JDK) which have led to creation of error-revealing tests and identification of real bugs, it has not been investigated in the literature how useful are the regression test suites generated by Randoop. In this paper, we have investigated flakiness and fragility of Randoop's regression tests during evolution of 5 open source Java projects with a total of 78 versions. The results demonstrate that the flakiness of the regression tests is not generally noticeable, since in our dataset, only 5% of the classes have at least one flaky regression tests. In addition, test fragility analysis reveals that in most versions of the projects under study, the regression tests generated by Randoop could be successfully executed on many of later versions. Actually, for 2 out of 5 projects in the experiments, the regression tests generated for each version could be successfully executed on all the later versions of the project.

show abstract

GRT: Program-Analysis-Guided Random Testing (T)

Cited by 56 publications

References 56 publications

Classification of Randomly Generated Test Cases

Classification of Randomly Generated Test Cases

JUGE: An infrastructure for benchmarking Java unit test generators

An Experimental Study on Flakiness and Fragility of Randoop Regression Test Suites

Contact Info

Product

Resources

About