A benchmark-based evaluation of search-based crash reproduction

Abstract

Crash reproduction approaches help developers during debugging by generating a test case that reproduces a given crash. Several solutions have been proposed to automate this task. However, the proposed solutions have been evaluated on a limited number of projects, making comparison difficult. In this paper, we enhance this line of research by proposing JCrashPack, an extensible benchmark for Java crash reproduction, together with ExRunner, a tool to simply and systematically run evaluations. JCrashPack contains 200 stack traces from various Java projects, including industrial open source ones, on which we run an extensive evaluation of EvoCrash, the state-of-the-art tool for search-based crash reproduction. EvoCrash successfully reproduced 43% of the crashes. Furthermore, we observed that reproducing NullPointerException, IllegalArgumentException, and IllegalStateException is relatively easier than reproducing ClassCastException, ArrayIndexOutOfBoundsException and StringIndexOutOfBoundsException. Our results include a detailed manual analysis of EvoCrash outputs, from which we derive 14 current challenges for crash reproduction, among which the generation of input data and the handling of abstract and anonymous classes are the most frequents. Finally, based on those challenges, we discuss future research directions for search-based crash reproduction for Java.

Search‐based crash reproduction using behavioural model seeding

“…In some executions, the initialized population is closer to the objectives, and the search process can achieve reproduction faster. Similar variations are reported in the JCrashPack empirical evaluation as well . According to the reported standard deviations, we can see that this value increases for all of the configurations of test seeding compared with no seeding.…”

“…Relying on the EvoCrash experience , we developed Botsing , a framework for crash reproduction with extensibility in mind. Botsing also relies on EvoSuite for the code instrumentation during test generation and execution by using evosuite‐client as a dependency.…”

“…In a recent study about the evaluation of search‐based crash reproduction approaches, Soltani et al introduced a new benchmark, called JCrashPack , containing 200 real‐world crashes from seven projects: JFreeChart , a framework for creating interactive charts; Commons‐lang , a library providing additional utilities to the java.lang API; Commons‐math , a library of mathematics and statistics components; Mockito , a testing framework for object mocking; Joda‐time , a library for date and time manipulation; XWiki , a popular enterprise wiki management system; and ElasticSearch , a distributed RESTful search and analytics engine. We use the same benchmark for the empirical evaluation of model seeding and test seeding on crash reproduction.…”

It Is Not Only About Control Dependent Nodes: Basic Block Coverage for Search-Based Crash Reproduction

“…Crash selection. We select crashes from JCrashPack [17], a benchmark containing hard-to-reproduce Java crashes. We apply the two fitness functions with and without using BBC as a secondary objective to 124 crashes, which have also been used in a recent study [8].…”

“…Method fromMap from XWIKI version 8.1[17] p u b l i c B a s e C o l l e c t i o n fromMap ( Map < [ . .…”