Syntactic Vs. Semantic similarity of Artificial and Real Faults in Mutation Testing Studies

Ojdanić, Miloš; Garg, Aayush; Khanfir, Ahmed; Degiovanni, Renzo; Papadakis, Mike; Traon, Yves Le

doi:10.48550/arxiv.2112.14508

Cited by 2 publications

(3 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multiple studies have been focused on the relationship between artificial and real faults [47]. The results of the studies conducted by Ojdanic et al [45], Papadakis et al [49], Just et al [30] and Andrews et al [9] showed that there is a correlation between tests broken by a bug and tests killing mutants. Meaning that artificial faults can be used as alternatives to real faults in controlled studies.…”

Section: Related Workmentioning

confidence: 99%

Efficient Mutation Testing via Pre-Trained Language Models

Khanfir¹,

Degiovanni²,

Papadakis³

et al. 2023

Preprint

View full text Add to dashboard Cite

Mutation testing is an established fault-based testing technique. It operates by seeding faults into the programs under test and asking developers to write tests that reveal these faults. These tests have the potential to reveal a large number of faults -those that couple with the seeded ones -and thus are deemed important. To this end, mutation testing should seed faults that are both "natural" in a sense easily understood by developers and strong (have high chances to reveal faults). To achieve this we propose using pre-trained generative language models (i.e. CodeBERT) that have the ability to produce developer-like code that operates similarly, but not exactly, as the target code. This means that the models have the ability to seed natural faults, thereby offering opportunities to perform mutation testing. We realise this idea by implementing µBERT, a mutation testing technique that performs mutation testing using CodeBert and empirically evaluated it using 689 faulty program versions. Our results show that the fault revelation ability of µBERT is higher than that of a state-of-the-art mutation testing (PiTest), yielding tests that have up to 17% higher fault detection potential than that of PiTest. Moreover, we observe that µBERT can complement PiTest, being able to detect 47 bugs missed by PiTest, while at the same time, PiTest can find 13 bugs missed by µBERT.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient Mutation Testing via Pre-Trained Language Models

Khanfir¹,

Degiovanni²,

Papadakis³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…This is a pre-trained model for Programming Languages. The usage of this model in the fault injection domain shows its ability to seed "natural" faults that, we can say, semantically resemble real faults [34], [35]. Effectively, the faults injected resemble what a real programmer could write (regarding the programmatic rules, convention, etc) [36].…”

Section: Nlp For Fault Injection (Not Vulnerability Injection)mentioning

confidence: 99%

“…CodeBERT has also been applied for mutation testing [35], [36]. For this purpose, researchers used the Masked Language Modeling task that takes a sentence with one masked token and the goal of the model is to find the most likely tokens to replace it.…”

Section: Nlp For Fault Injection (Not Vulnerability Injection)mentioning

confidence: 99%

IntJect: Vulnerability Intent Bug Seeding

Petit

Khanfir

Soremekun

et al. 2022

2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)

View full text Add to dashboard Cite

Studying and exposing software vulnerabilities is important to ensure software security, safety, and reliability. Software engineers often inject vulnerabilities into their programs to test the reliability of their test suites, vulnerability detectors, and security measures. However, state-of-the-art vulnerability injection methods only capture code syntax/patterns, they do not learn the intent of the vulnerability and are limited to the syntax of the original dataset. To address this challenge, we propose the first intent-based vulnerability injection method that learns both the program syntax and vulnerability intent. Our approach applies a combination of NLP methods and semantic-preserving program mutations (at the bytecode level) to inject code vulnerabilities. Given a dataset of known vulnerabilities (containing benign and vulnerable code pairs), our approach proceeds by employing semantic-preserving program mutations to transform the existing dataset to semantically similar code. Then, it learns the intent of the vulnerability via neural machine translation (Seq2Seq) models. The key insight is to employ Seq2Seq to learn the intent (context) of the vulnerable code in a manner that is agnostic of the specific program instance. We evaluate the performance of our approach using 1275 vulnerabilities belonging to five (5) CWEs from the Juliet test suite. We examine the effectiveness of our approach in producing compilable and vulnerable code. Our results show that INTJECT is effective, almost all (99%) of the code produced by our approach is vulnerable and compilable. We also demonstrate that the vulnerable programs generated by INTJECT are semantically similar to the withheld original vulnerable code. Finally, we show that our mutation-based data transformation approach outperforms its alternatives, namely data obfuscation and using the original data.

show abstract

Syntactic Vs. Semantic similarity of Artificial and Real Faults in Mutation Testing Studies

Cited by 2 publications

References 32 publications

Efficient Mutation Testing via Pre-Trained Language Models

Efficient Mutation Testing via Pre-Trained Language Models

IntJect: Vulnerability Intent Bug Seeding

Contact Info

Product

Resources

About