Investigating the Effectiveness of Mutation Testing Tools in the Context of Deep Neural Networks

Software Verification and Validation of Safe Autonomous Cars: A Systematic Literature Review

“…According to experiments in recent studies, another criticism towards the framework and usage of neuron coverage in general has emerged. Reference [58] notes that even achieving 100% neuron coverage is not enough to verify the safety of DNN applications as maximum neuron coverage is proved to be reached by merely using specific input vectors from the training data.…”

“…It allows engineers to have a metric that corresponds to the ratio of identified vs missed mutations, in order to evaluate the adequacy of test suites [4]. Mutants are identified if the output of the mutated software is different than that of the original one [58]. Reference [4] states that mutation testing is an efficient method to mock real faults.…”

“…Reference [58] focuses on assessing the effectiveness of the current mutation tools for DNN. During the tests in the study, a threshold (1%) is set to check the deviations in the accuracy of decisions of DNN algorithms.…”

“…During the tests in the study, a threshold (1%) is set to check the deviations in the accuracy of decisions of DNN algorithms. Reference [58] demonstrates that it is possible to detect mutants with a deviation that exceeds the threshold and there was no case that no mutant was detected. To show the effectiveness of the tests a criterion called mutation score, which is the ratio of detected mutants to the remaining mutants, is used.…”

“…To show the effectiveness of the tests a criterion called mutation score, which is the ratio of detected mutants to the remaining mutants, is used. Reference [58] notes that such criterion for the quality of test suits is important to show that safety-critical systems such as autonomous cars meet the Modified Condition/Decision Coverage criterion as recommended in ISO 26262. Experiments in the study show that the mutation score of experimented algorithms varies from 40% to 65%.…”