DeepMutation++: A Mutation Testing Framework for Deep Learning Systems

Hu, Qiang; Ма, Лей; Xie, Xiaofei; Yu, Bing; Liu, Yang; Zhao, Jianjun

doi:10.1109/ase.2019.00126

Cited by 76 publications

(42 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The mutants, the nodes of which have no incoming edges, are non-redundant (or dominant), while the remaining mutants are redundant. In order to assess the performance of the pre-training mutation operators proposed and implemented in this paper, we compare DEEPCRIME to DeepMutation++ [18], a model-level, post-training mutation testing tool that implements 8 operators from the ones introduced by Ma et al [29] and 9 new operators designed for stateful recurrent neural networks. We measure the sensitivity of DEEPCRIME and DeepMutation++, defined as the relative variation of the mutation score when moving from a weak to a strong test set:…”

Section: Rq2 [Redundant Mutation Operators]mentioning

confidence: 99%

“…Availability of a mutation tool that can inject changes imitating real faults would be extremely useful also for these approaches. There exists a number of DL specific mutation operators proposed in the literature [29,33] and 8 of them are implemented in the tool DeepMutation++ [18], which manipulates a pre-trained model to produce its mutant versions. However, none of the existing DL mutation operators, including those in DeepMutation++, are based on real faults that affect DL systems.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

DeepCrime: mutation testing of deep learning systems based on real faults

Humbatova

Jahangirova

Tonella

2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Deep Learning (DL) solutions are increasingly adopted, but how to test them remains a major open research problem. Existing and new testing techniques have been proposed for and adapted to DL systems, including mutation testing. However, no approach has investigated the possibility to simulate the effects of real DL faults by means of mutation operators. We have defined 35 DL mutation operators relying on 3 empirical studies about real faults in DL systems. We followed a systematic process to extract the mutation operators from the existing fault taxonomies, with a formal phase of conflict resolution in case of disagreement. We have implemented 24 of these DL mutation operators into DEEPCRIME, the first source-level pre-training mutation tool based on real DL faults. We have assessed our mutation operators to understand their characteristics: whether they produce interesting, i.e., killable but not trivial, mutations. Then, we have compared the sensitivity of our tool to the changes in the quality of test data with that of DeepMutation++, an existing post-training DL mutation tool.

show abstract

Section: Rq2 [Redundant Mutation Operators]mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

DeepCrime: mutation testing of deep learning systems based on real faults

Humbatova

Jahangirova

Tonella

2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

show abstract

“…ratio parameter that identifies what ratio of weights/neurons is affected by the mutation. We used three values of ratio equal to 0.01, 0.03, and 0.05 for each mutation operator [22]. The remaining three operators are applied to layers.…”

Section: Resultsmentioning

confidence: 99%

“…In order to obtain models with degraded driving performance, we created mutants of the original model by means of the DL mutation tool DeepMutation++ [22]. Specifically, we used DeepMutation++'s Gaussian Fuzzing mutation operator with ratio=0.03 for Lake Track and Jungle Track, and the Neuron Effect Block operator with ratio=0.03 for Mountain Track, to obtain representatives of poor driving quality models.…”

Section: Procedures and Metrics (Rq1)mentioning

confidence: 99%

See 1 more Smart Citation

Quality Metrics and Oracles for Autonomous Vehicles Testing

Jahangirova

Stocco

Tonella

2021

2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST)

View full text Add to dashboard Cite

The race for deploying AI-enabled autonomous vehicles (AVs) on public roads is based on the promise that such self-driving cars will be as safe as or safer than human drivers. Numerous techniques have been proposed to test AVs, which however lack oracle definitions that account for the quality of driving, due to the lack of a commonly used set of metrics. Towards filling this gap, we first performed a systematic analysis of the literature concerning the assessment of the quality of driving of human drivers and extracted 126 metrics. Then, we measured the correlation between such metrics and the human perception of driving quality when AVs are driving. Lastly, we performed a study based on mutation analysis to assess whether the 26 metrics that best capture the quality of AV driving according to the human study can be used as functional oracles. Our results, targeting the Udacity platform, indicate that our automated oracles can kill a high proportion of mutants at a zero or very low false alarm rate, and therefore can be used as effective functional oracles for the quality of driving of AVs.

show abstract

Equivalent mutant identification using hybrid wavelet convolutional rain optimization

Jammalamadaka

Parveen

2021

Softw Pract Exp

View full text Add to dashboard Cite

Mutation testing is a significant software testing approach that identifies the faults present in the source codes and the potentiality of the test cases in detecting the mutated codes. In this testing approach, equivalent mutant detection is complicated as the test cases will not identify the mutated codes from the source program. To overcome this problem, the article proposes a novel hybrid strategy known as the hybrid wavelet convolutional rain optimization (HWCRO) to classify the equivalent mutants present in the source codes accurately. The proposed technique considers three different classes of equivalent mutants based on the RIPR model and exactly identifies the mutated code. Initially, the features such as the semantic similarity and the information entropy are extracted, and these features are given as the input to the wavelet convolutional neural network (wCNN) classifier. The dimensions of the features are reduced in the convolutional layers using the wavelet function, which enhances the classifier's performance. To improve the classification accuracy, the loss function is minimized with an adaptive rain optimization algorithm (ROA) that iteratively tunes the parameter of wCNN. The proposed approach is compared with the existing classification techniques based on the parameters such as precision, recall, f1-score, and accuracy, and the simulation results yielded 85.17% accuracy value for the proposed approach.

show abstract

DeepMutation++: A Mutation Testing Framework for Deep Learning Systems

Cited by 76 publications

References 8 publications

DeepCrime: mutation testing of deep learning systems based on real faults

DeepCrime: mutation testing of deep learning systems based on real faults

Quality Metrics and Oracles for Autonomous Vehicles Testing

Equivalent mutant identification using hybrid wavelet convolutional rain optimization

Contact Info

Product

Resources

About