Automatic repair of real bugs in java: a large-scale experiment on the defects4j dataset

Abstract

International audienceDefects4J is a large, peer-reviewed, structured dataset of real-world Java bugs. Each bug in Defects4J comes with a test suite and at least one failing test case that triggers the bug. In this paper, we report on an experiment to explore the effectiveness of automatic test-suite based repair on Defects4J. The result of our experiment shows that the considered state-of-the-art repair methods can generate patches for 47 out of 224 bugs. However, those patches are only test-suite adequate, which means that they pass the test suite and may potentially be incorrect beyond the test-suite satisfaction correctness criterion. We have manually analyzed 84 different patches to assess their real correctness. In total, 9 real Java bugs can be correctly repaired with test-suite based repair. This analysis shows that test-suite based repair suffers from under-specified bugs, for which trivial or incorrect patches still pass the test suite. With respect to practical applicability, it takes on average 14.8 minutes to find a patch. The experiment was done on a scientific grid, totaling 17.6 days of computation time. All the repair systems and experimental results are publicly available on Github in order to facilitate future research on automatic repair

Contract-based program repair without the contracts

“…To ascertain which candidates are suitable, a fix validation process, which follows fix generation, runs all tests T that exercise the faulty method fixMe against each generated candidate fix. Candidate fixes that pass all tests T are classified as valid (also "test-suite adequate" [19]) and retained; other candidates, which fail some tests, are discarded-as they do not fix the fault, they introduce a regression, or both.…”

“…We quantitatively compare JAID to all other available tools for APR of Java programs that have also used DEFECTS4J in their evaluations: 4 1) jGenProg is the implementation of GenProg [14], [33]-which works on C-for Java programs; we refer to jGenProg's evaluation in [19]; 2) jKali is the implementation of Kali [28]-which works on C-for Java programs; we refer to jKali's evaluation in [19]; 3) Nopol focuses on fixing Java conditional expression; we refer to Nopol's evaluation in [19]; 4) xPAR is a reimplementation of PAR [12]-which is not publicly available-discussed in [13] and [35]; 5) HDA implements the "history-driven" technique of [13]; 6) ACS implements the "precise condition synthesis" of [35].…”

“…Since validation is against a finite-often small-number of tests, there is no guarantee that a valid repair is genuinely correct against a complete, and implicit, specification of the method. Indeed, experiments have repeatedly confirmed [19], [28], [29] that automated program repair techniques are prone to producing a significant fraction of valid but incorrect repairs, which merely happen to pass all available tests but are clearly inadequate from a programmer's perspective.…”

A Survey of Test Based Automatic Program Repair

“…Researchers tried to promote their generalization by conducting their experiments on large real programs [54] with real bugs reported and compared the machine generated patches with human made ones [30]. Comparisons among existing repair systems rely on same benchmarks, Defects4J [56], ManyBugs [80]and Codeflaws [1] could be considerable alternatives.…”

Towards the Quality Improvement of Cross-Platform Mobile Applications

“…Their results showed the adoption of crossplatform frameworks as development tools always implies an 15 http://junit.org/junit4/ 16 https://github.com/dotnet/roslyn 17 https://developer.xamarin.com/guides/testcloud/uitest/ increase in energy consumption, even if the final application is a real native application.…”