Identifying test-suite-overfitted patches through test case generation

Qi, Xin; Reiss, Steven P.

doi:10.1145/3092703.3092718

Cited by 91 publications

(84 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study focuses on test-suite adequate patches, which means that the generated patches make the test suite pass; yet, there is no guarantee that they fix the bugs. Studying patch correctness [19,44,49] is out of the scope of this work. Our goal is to analyze the current state of the automatic program repair tools and identify potential flaws and improvements.…”

Section: Discussionmentioning

confidence: 99%

Empirical review of Java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts

Durieux

Madeiral

Martínez

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

117

124

View full text Add to dashboard Cite

In the past decade, research on test-suite-based automatic program repair has grown significantly. Each year, new approaches and implementations are featured in major software engineering venues. However, most of those approaches are evaluated on a single benchmark of bugs, which are also rarely reproduced by other researchers. In this paper, we present a large-scale experiment using 11 Java test-suite-based repair tools and 5 benchmarks of bugs. Our goal is to have a better understanding of the current state of automatic program repair tools on a large diversity of benchmarks. Our investigation is guided by the hypothesis that the repairability of repair tools might not be generalized across different benchmarks of bugs. We found that the 11 tools 1) are able to generate patches for 21% of the bugs from the 5 benchmarks, and 2) have better performance on Defects4J compared to other benchmarks, by generating patches for 47% of the bugs from Defects4J compared to 10-30% of bugs from the other benchmarks. Our experiment comprises 23,551 repair attempts in total, which we used to find the causes of non-patch generation. These causes are reported in this paper, which can help repair tool designers to improve their techniques and tools.

show abstract

Section: Discussionmentioning

confidence: 99%

Empirical review of Java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts

Durieux

Madeiral

Martínez

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

117

124

View full text Add to dashboard Cite

show abstract

“…For example, Smith et al [31] use general-purpose automatic test generation tool such as KLEE [43] to generate test suites for C language. For Java language, Xin et al [34] propose Dif-fTGen, a test generation tool specially designed to generate tests that can identify incorrect patches generated by APR techniques. DiffTGen attempts to generate test cases that cover the syntactic and semantic differences between the generated patch and the developer-provided patch.…”

Section: B Validation Of Apr-generated Patchesmentioning

confidence: 99%

“…If the patch does not pass the validation, the second and third steps will be repeated until a valid patch is found or a predefined limitation is reached, e.g., the execution time. Over the years, many studies have been conducted with the aim to better identify the fault location [10][11][12][13][14][15][16][17][18][19], advance the patch generation process [2][3][4][5][6][7][8][9][20][21][22][23][24][25][26][27][28][29], and enhance the assessment of patch correctness [30][31][32][33][34]. The scope of this paper belongs to the last one.…”

Section: Introductionmentioning

confidence: 99%

“…However, this method is biased and inefficient as pointed out by the study [36] that the test suites in real world systems are usually weak such that most of the patches that pass all tests are incorrect. This problem, which is often referred to as patch overfitting [31,32,34], motivates the need of new methodologies for patch correctness assessment. To address this concern, recent works mainly follow two methods for evaluating patch correctness.…”

Section: Introductionmentioning

confidence: 99%

“…To address this concern, recent works mainly follow two methods for evaluating patch correctness. One is utilizing an independent test suite generated by automatic test generation tools to verify the patch correctness [24,31,34,37]. Following this method, a patch is labeled as correct if it passes both the original associated test suites and the newly generated one.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

How Different Is It Between Machine-Generated and Developer-Provided Patches? : An Empirical Study on the Correct Patches Generated by Automated Program Repair Techniques

Wang

Wen

Chen

et al. 2019

2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

View full text Add to dashboard Cite

Background: Over the years, Automated Program Repair (APR) has attracted much attention from both academia and industry since it can reduce the costs in fixing bugs. However, how to assess the patch correctness remains to be an open challenge. Two widely adopted ways to approach this challenge, including manually checking and validating using automated generated tests, are biased (i.e., suffering from subjectivity and low precision respectively). Aim: To address this concern, we propose to conduct an empirical study towards understanding the correct patches that are generated by existing state-of-the-art APR techniques, aiming at providing guidelines for future assessment of patches. Method: To this end, we first present a Literature Review (LR) on the reported correct patches generated by recent techniques on the Defects4J benchmark and collect 177 correct patches after a process of sanity check. We investigate how these machine-generated correct patches achieve semantic equivalence, but syntactic difference compared with developerprovided ones, how these patches distribute in different projects and APR techniques, and how the characteristics of a bug affect the patches generated for it. Results: Our main findings include 1) we do not need to fix bugs exactly like how developers do since we observe that 25.4% (45/177) of the correct patches generated by APR techniques are syntactically different from developerprovided ones; 2) the distribution of machine-generated correct patches diverges for the aspects of Defects4J projects and APR techniques; and 3) APR techniques tend to generate patches that are different from those by developers for bugs with large patch sizes. Conclusion: Our study not only verifies the conclusions from previous studies but also highlights implications for future study towards assessing patch correctness. Keywords-Automated Program Repair; Defects4J; patch correctness assessment.RQ1 How do machine-generated correct patches differ from developer-provided ones?RQ2 How do different types of patches distribute? RQ3 Do APR tools tend to generate correct patches but different from the developer-provided ones for bugs with certain characteristics?A patch is generated based on the buggy location identified by fault localization techniques (i.e., denoted as edit point in this study) with certain code modifications. Based on this, the differences between patches can be distinguished in terms of two aspects, edits points and code modifications. To answer RQ1, we compare the collected patches with developerprovided ones and classify them into four types based on the aforementioned two aspects. We further investigate how the patches that are syntactically different from developer-provided ones achieve semantic equivalence. In RQ2, we investigate the distribution of patches from two aspects (i.e., different De-fects4J projects and APR techniques) and observe that fault localization is critical for generating correct patches for bugs in three projects of Defects4J. In RQ3, we aim at investigating whethe...

show abstract

Enhanced evolutionary automated program repair by finer‐granularity ingredients and better search algorithms

Wang,

Liu,

Lin

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

SummaryBug repair is time consuming and tedious, which hampers software maintenance. To alleviate the burden, automated program repair (APR) is proposed and has been fruitful in the last decade. Evolutionary repair is the seminal work of this field and proliferated a family of approaches. The performance of evolutionary repair approaches is affected by two main factors: (1) search space, which defines all possible patches, and (2) search algorithms, which navigate the space. Although recent approaches have achieved remarkable progress, the main challenges of the two factors still remain. On one hand, the different kinds of search space are very coarse for containing correct patches. On the other hand, the search process guided by genetic algorithms is inefficient in finding the correct patches in an appropriate time budget. In this paper, we propose MicroRepair, a new evolutionary repair approach to address the two challenges. Rather than finding statement‐level patches like existing genetic repair approaches, MicroRepair enlarges the search space by breaking the statements into finer‐granularity ingredients that consist of AST leaves. As the search space grows exponentially, the former search algorithms may become inefficient in navigating the larger space. We utilize the best multiobjective search algorithm selected from our empirical comparison of a set of search algorithms. At last, we find redundancies search in the existing genetic process, and we further design a history‐aware search strategy to accelerate the process.We evaluated MicroRepair on 224 bugs of real‐world from the benchmark Defects4J and compared it with several state‐of‐the‐art repair approaches. The evaluation results show that MicroRepair correctly repaired 26 bugs with a precision of 62%, which significantly outperforms the state‐of‐the‐art evolutionary APR approaches in terms of precision. Moreover, the history‐aware search boosts the repair execution speed by 4% on average.

show abstract

Identifying test-suite-overfitted patches through test case generation

Cited by 91 publications

References 34 publications

Empirical review of Java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts

Empirical review of Java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts

How Different Is It Between Machine-Generated and Developer-Provided Patches? : An Empirical Study on the Correct Patches Generated by Automated Program Repair Techniques

Enhanced evolutionary automated program repair by finer‐granularity ingredients and better search algorithms

Contact Info

Product

Resources

About