The strength of random search on automated program repair

Abstract

Automated program repair recently received considerable attentions, and many techniques on this research area have been proposed. Among them, two genetic-programmingbased techniques, GenProg and Par, have shown the promising results. In particular, GenProg has been used as the baseline technique to check the repair effectiveness of new techniques in much literature. Although GenProg and Par have shown their strong ability of fixing real-life bugs in nontrivial programs, to what extent GenProg and Par can benefit from genetic programming, used by them to guide the patch search process, is still unknown.To address the question, we present a new automated repair technique using random search, which is commonly considered much simpler than genetic programming, and implement a prototype tool called RSRepair. Experiment on 7 programs with 24 versions shipping with real-life bugs suggests that RSRepair, in most cases (23/24), outperforms GenProg in terms of both repair effectiveness (requiring fewer patch trials) and efficiency (requiring fewer test case executions), justifying the stronger strength of random search over genetic programming. According to experimental results, we suggest that every proposed technique using optimization algorithm should check its effectiveness by comparing it with random search.

A Survey of Test Based Automatic Program Repair

“…They implemented TrpAutoRepair [19] and RSRepair [20], which use the same mutation approaches as GenProg and run prioritized test cases to validate variants hoping a useless variant could be detected as soon as possible. Distinct from GenProg, no evolving means no use to execute all the tests to calculate the fitness.…”

“…RSRepair stops patch validation regression as far as a test case fails, which reduces the time expense significantly. Experiment reports that both the patch trials and test case executions per validated fix of RSRepair are less than those of GenProg, which indicates that the benefit brought by genetic programming cannot make up the cost caused by fitness evaluations, and thus worsening the patch search process [20]. On the contrary, Assiri and Bieman [21] provided MUT-APR to illustrate the strength of operator mutation against using the existing code to generate patch while keeping the genetic fitness evaluation in patch selection.…”

“…For this reason, test case redundancy elimination, and some strategies such as test cases prioritization can help terminate the regressive testing as soon as possible. Qi et al [20] illustrated the strength of most-failing-cases-executed-first in performance. We believe that the properly designed validating strategy can harmonize the contradictions between coverage and performance to a considerable degree.…”

Un-preprocessing

“…This popularity of CPP causes a difficulty in program modification tools, such as program-repair tools [20,21,31,36] and API evolution tools [23,30,35], where direct modification of source code is involved. Such tools typically do not handle C preprocessor directives.…”

“…Such tools typically do not handle C preprocessor directives. This is not only in the case for casual uses, where the target languages bear no connection to C. We have investigated the implementations of three influential bug-fixing tools on the C programming language: GenProg [20,21], RSRepair [36], and SemFix [31], and all of them work only on preprocessed code. Users have to manually inspect the preprocessed code, and copy the changes to the original code-risking of introducing bugs in the process.…”