iBiR : Bug-report-driven Fault Injection

Abstract

Much research on software engineering relies on experimental studies based on fault injection. Fault injection, however, is not often relevant to emulate real-world software faults since it “blindly” injects large numbers of faults. It remains indeed challenging to inject few but realistic faults that target a particular functionality in a program. In this work, we introduce iBiR , a fault injection tool that addresses this challenge by exploring change patterns associated to user-reported faults. To inject realistic faults, we create mutants by re-targeting a bug report driven automated program repair system, i.e., reversing its code transformation templates. iBiR is further appealing in practice since it requires deep knowledge of neither code nor tests, but just of the program’s relevant bug reports. Thus, our approach focuses the fault injection on the feature targeted by the bug report. We assess iBiR by considering the Defects4J dataset. Experimental results show that our approach outperforms the fault injection performed by traditional mutation testing in terms of semantic similarity with the original bug, when applied at either system or class levels of granularity, and provides better, statistically significant, estimations of test effectiveness (fault detection). Additionally, when injecting 100 faults, iBiR injects faults that couple with the real ones in around 36% of the cases, while mutation testing achieves less than 4% .

Efficient Mutation Testing via Pre-Trained Language Models

“…Their results are promising, however, the fact that these techniques depend on the availability of numerous, diverse, comprehensive and untangled fix commits [27] of not coupled faults [43], which is often hard to fulfil in practice, may hinder their performance. Acknowledging for the injection location [13], [42], Khanfir et al [32] combined the usage of information retrieved from bug reports with inverted automated-program-repair patterns to replicate real faults fixable by the original fix-patterns. Their results showed that they can generate faults that mimic real ones, however, their approach remains dependent and limited to the presence of good bug reports.…”

“…Similarly, Mirshokraie et al [41] compute complexity metrics from program executions to extract loca-tions with good observability to mutate. Other approaches restrict the fault injection on specific locations of the program, such as the code impacted by the last commits [38], [58] for better usability in continuous integration, or targeting locations related to a given bug-report [32] to target a specific feature or behaviour, etc. More recent advances have resulted in powerful techniques for cost-effectively selecting mutants, i.e., by avoiding the analysis of redundant mutants (basically, equivalent and subsumed ones) [24], [25], [28].…”

“…Recent research [32] has shown that some real faults are only captured by using complex patterns, i.e. patterns that require more than one token mutation.…”

IntJect: Vulnerability Intent Bug Seeding

“…The vulnerability intent is predefined rather than learnt as INT-JECT does. In contrast, IBIR exploits bug reports to identify source code locations to be made faulty and inverts program repair fixes to inject faults [13]. Notably, both BUG-INJECTOR and IBIR do not specifically target vulnerabilities.…”

On Comparing Mutation Testing Tools through Learning-based Mutant Selection

“…Most approaches inject faults based on predefined syntactic transformation rules (aka mutation operators) [6], [18], such as replacing an instance of a relational operator with another operator, e.g., replacing > with >=. Other approaches aim at injecting faults by either following fault patterns created or learned from recurrent fault instances [29], [51], [48] or by employing code pretrained language models [19]. These approaches have been implemented and made openly available as tools, serving the main purposes of mutation testing -tests assessment and guidance criterion.…”