Alleviating patch overfitting with automatic test generation: a study of feasibility and effectiveness for the Nopol repair system

Abstract

Among the many different kinds of program repair techniques, one widely studied family of techniques is called test suite based repair. However, test suites are in essence input-output specifications and are thus typically inadequate for completely specifying the expected behavior of the program under repair. Consequently, the patches generated by test suite based repair techniques can just overfit to the used test suite, and fail to generalize to other tests. We deeply analyze the overfitting problem in program repair and give a classification of this problem. This classification will help the community to better understand and design techniques to defeat the overfitting problem. We further propose and evaluate an approach called UnsatGuided, which aims to alleviate the overfitting problem for synthesis-based repair techniques with automatic test case generation. The approach uses additional automatically generated tests to strengthen the repair constraint used by synthesis-based repair techniques. We analyze the effectiveness of UnsatGuided: 1) analytically with respect to alleviating two different kinds of overfitting issues; 2) empirically based on an experiment over the 224 bugs of the Defects4J repository. The main result is that automatic test generation is effective in alleviating one kind of overfitting issue-regression introduction, but due to oracle problem, has minimal positive impact on alleviating the other kind of overfitting issue-incomplete fixing.

A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark

“…To sum up, our contributions are: • A new version of QuixBugs that is usable for automatic repair research on Java programs, together with extensive data about the characteristics of QuixBugs. • The confirmation of 2 empirical facts of program repair, improving their external validity: 1) the state-of-the-art program repair tools produce overfitting patches, this confirms the results of [35], [31], [15]; 2) the state-ofthe-art program repair tools also produce correct patches [29], [20]; 3) automatically generated tests can help to assess the correctness of patches in scientific studies, this confirms the results of [40], [36], [41]. • Three new and important findings about program repair: 1) the state-of-the-art program repair tools are able to repair programs with only failing test cases and no passing tests at all; 2) it is useful to design program specific test generators to discard incorrect patches; and 3) a small number of automatically generated test cases is enough to identify incorrect patches in scientific studies.…”

“…In our experiment, we consider three techniques for patch correctness assessment: a) using automatically generated tests by a search-based approach based on a reference version [41]; b) using automatically generated tests by a program specific generator based on a reference version [2]; and c) manual analysis of patch correctness [20]. a) Search-based Test Generation Technique: Using automated test generation is one way for assessing patch correctness [37], [36], [41], [40]. In our study, the search-based test generator technique takes as input a reference version of buggy program.…”

“…Eventually, we obtain n different independent JUnit test suites for each program. Since Evosuite is a randomized algorithm, we take n = 30 for the best practice [41]. We further remove those generated tests that fail on the reference version (due to limitation of Evosuite).…”

“…On the contrary, a "correct" patch means that it is not overfitting to the input data and to the considered test cases. To evaluate the correctness of patches, we use three different patch correctness techniques based on: 1) a search-based test generation tool [41]; 2) a custom program specific test generation tool [2]; and 3) manual analysis [20]. In RQ4, we analyze in detail the effectiveness of automated patch correctness assessment techniques used in RQ3.…”

Empirical review of Java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts

“…This study focuses on test-suite adequate patches, which means that the generated patches make the test suite pass; yet, there is no guarantee that they fix the bugs. Studying patch correctness [19,44,49] is out of the scope of this work. Our goal is to analyze the current state of the automatic program repair tools and identify potential flaws and improvements.…”