A critical review on the evaluation of automated program repair systems

Abstract

Automated Program Repair (APR) has attracted significant attention from software engineering research and practice communities in the last decade. Several teams have recorded promising performance in fixing real bugs and there is a race in the literature to fix as many bugs as possible from established benchmarks. Gradually, repair performance of APR tools in the literature has gone from being evaluated with a metric on the number of generated plausible patches to the number of correct patches. This evolution is necessary after a study highlighting the overfitting issue in test suite-based automatic patch generation. Simultaneously, some researchers are also insisting on providing time cost in the repair scenario as a metric for comparing state-of-the-art systems. In this paper, we discuss how the latest evaluation metrics of APR systems could be biased. Since design decisions (both in approach and evaluation setup) are not always fully disclosed, the impact on repair performance is unknown and computed metrics are often misleading. To reduce notable biases of design decisions in program repair approaches, we conduct a critical review on the evaluation of patch generation systems and propose eight evaluation metrics for fairly assessing the performance of APR tools. Eventually, we show with experimental data on 11 baseline program repair systems that the proposed metrics allow to highlight some caveats in the literature. We expect wide adoption of these metrics in the community to contribute to boosting the development of practical, and reliably performable program repair tools.

Revisiting Test Cases to Boost Generate-and-Validate Program Repair

“…Then, we present the experimental setup to answer these research questions. [9]. In previous studies, a few APR systems have been proposed for addressing the specific types of bugs (e.g., NPEFix [43] and VFix [44] for null pointer dereferences).…”

“…Therefore, the promise of automated program repair (APR) to alleviate the manual burden is appealing [6]. APR has thus been a prolific research field in the last decade [7], [8], [9]. Among the various approaches that are proposed, many fall under the category of generate-and-validate APR [10], [11], [12], [13], [14], [15], [16], [17] where program faults are localized to drive patch generation before patch validation.…”

A Comparative Performance Analysis of Computational Intelligence Techniques to Solve the Asymmetric Travelling Salesman Problem

“…Model-Induced Max-Min Ant Colony Optimization (MIMM-ACO) [ 63 ] is a hybridization of the Max-Min Ant System algorithm with Karp's Patching technique [ 64 ] and the Patch heuristics [ 65 ] resulting in two main adjustments to the classical Max-Min Ant System algorithm (MMAS). First, the static pheromone weighting system is replaced by a dynamic pheromone weighting mechanism.…”

Incremental Formula-Based Fix Localization

“…Automated program repair techniques usually start by using a fault localization or a fix localization [37,38] to identify a subset of code elements at which a patch can be applied. The effectiveness of the fix localization task is critical important to the effectiveness, as well as the reliability of automatic program repair [39,40]. The fix localization components in the semantic-based APR approaches (such as Angelix [37] and Nopol [36]) share the same objective with our approach that is finding angelic execution paths that make the failing test case to be a pass.…”