A probabilistic framework for mutation testing in deep neural networks

Tambon, Florian; Khomh, Foutse; Antoniol, Giuliano

doi:10.1016/j.infsof.2022.107129

Cited by 11 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Heuristically, we just assume it's a potential point of interest, better than any random point if we are constrained by the number of test environments to be generated and tested. Finally, Tambon et al [36] study more in-depth the effect that the choice of Deep Learning model instances has over the result of Mutation Testing in Supervised Learning using DeepCrime's approach. They showed that this choice can affect the outcome of the Mutation Testing and that using the Bayesian approach can mitigate this issue.…”

Section: Related Workmentioning

confidence: 99%

Mutation Testing of Deep Reinforcement Learning Based on Real Faults

Tambon¹,

Majdinasab²,

Nikanjam³

et al. 2023

Preprint

View full text Add to dashboard Cite

Testing Deep Learning (DL) systems is a complex task as they do not behave like traditional systems would, notably because of their stochastic nature. Nonetheless, being able to adapt existing testing techniques such as Mutation Testing (MT) to DL settings would greatly improve their potential verifiability. While some efforts have been made to extend MT to the Supervised Learning paradigm, little work has gone into extending it to Reinforcement Learning (RL) which is also an important component of the DL ecosystem but behaves very differently from SL. This paper builds on the existing approach of MT in order to propose a framework, RLMutation, for MT applied to RL. Notably, we use existing taxonomies of faults to build a set of mutation operators relevant to RL and use a simple heuristic to generate test cases for RL. This allows us to compare different mutation killing definitions based on existing approaches, as well as to analyze the behavior of the obtained mutation operators and their potential combinations called Higher Order Mutation(s) (HOM). We show that the design choice of the mutation killing definition can affect whether or not a mutation is killed as well as the generated test cases. Moreover, we found that even with a relatively small number of test cases and operators we manage to generate HOM with interesting properties which can enhance testing capability in RL systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Mutation Testing of Deep Reinforcement Learning Based on Real Faults

Tambon¹,

Majdinasab²,

Nikanjam³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Tambon et al [12] introduced Probabilistic Mutation Testing (PMT) for DNNs, considering the stochasticity during training. PMT provides consistent decisions on mutant killing and demonstrated effectiveness using three models and eight mutation operators.…”

Section: Mutationmentioning

confidence: 99%

“…Mutation testing involves making small modifications to the model, training data, or source code to reveal potential vulnerabilities and mistakes that may not be identified by conventional methods. Various approaches have been proposed for evaluating test suites of neural networks using novel mutation operators [10,11,12]. Coverage criteria are used to define important areas for test suite evaluation.…”

Section: Introductionmentioning

confidence: 99%

Deepcover: Defining Coverage Criteria for Test Suite Evaluation and Error Prediction Using State Machines Extraction from Rnns

Golshanrad

Faghih

2023

Preprint

View full text Add to dashboard Cite

“…Tambon et al (2023) describe silent bugs in popular deep learning frameworks that escape notice due to undetected error propagation.…”

mentioning

confidence: 99%

To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing

Gururaja,

Bertsch,

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

NLP is in a period of disruptive change that is impacting our methodologies, funding sources, and public perception. In this work, we seek to understand how to shape our future by better understanding our past. We study factors that shape NLP as a field, including culture, incentives, and infrastructure by conducting longform interviews with 26 NLP researchers of varying seniority, research area, institution, and social identity. Our interviewees identify cyclical patterns in the field, as well as new shifts without historical parallel, including changes in benchmark culture and software infrastructure. We complement this discussion with quantitative analysis of citation, authorship, and language use in the ACL Anthology over time. We conclude by discussing shared visions, concerns, and hopes for the future of NLP. We hope that this study of our field's past and present can prompt informed discussion of our community's implicit norms and more deliberate action to consciously shape the future.

show abstract

A probabilistic framework for mutation testing in deep neural networks

Cited by 11 publications

References 18 publications

Mutation Testing of Deep Reinforcement Learning Based on Real Faults

Mutation Testing of Deep Reinforcement Learning Based on Real Faults

Deepcover: Defining Coverage Criteria for Test Suite Evaluation and Error Prediction Using State Machines Extraction from Rnns

To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing

Contact Info

Product

Resources

About