Is neuron coverage a meaningful measure for testing deep neural networks?

Abstract

Recent effort to test deep learning systems has produced an intuitive and compelling test criterion called neuron coverage (NC), which resembles the notion of traditional code coverage. NC measures the proportion of neurons activated in a neural network and it is implicitly assumed that increasing NC improves the quality of a test suite. In an attempt to automatically generate a test suite that increases NC, we design a novel diversity promoting regularizer that can be plugged into existing adversarial attack algorithms. We then assess whether such attempts to increase NC could generate a test suite that (1) detects adversarial attacks successfully, (2) produces natural inputs, and (3) is unbiased to particular class predictions. Contrary to expectation, our extensive evaluation finds that increasing NC actually makes it harder to generate an effective test suite: higher neuron coverage leads to fewer defects detected, less natural inputs, and more biased prediction preferences. Our results invoke skepticism that increasing neuron coverage may not be a meaningful objective for generating tests for deep neural networks and call for a new test generation technique that considers defect detection, naturalness, and output impartiality in tandem. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Software reliability; • Computing methodologies → Neural networks.

RobOT: Robustness-Oriented Testing for Deep Learning Systems

“…Adversarial attacks often achieve higher robustness improvement than all three neuron coverage-guided fuzzing algorithms for simpler datasets such as MNIST, Fashion-MNIST and SVHN. This casts shadow on the usefulness of the test cases generated by neuron coverage-guided fuzzing algorithms in improving model robustness and is consistent with [6], [13], [23].…”

“…Along with the testing metrics, many test case generation algorithms are also proposed including gradient-guided perturbation [30], [46], black-box [42] and metric-guided fuzzing [12], [21], [43]. However, these testing works lack rigorous evaluation on their usefulness in improving the model robustness (although most of them claim so) and have been shown to be ineffective in multiple recent works [6], [13], [23]. Multiple metrics have been proposed in the machine learning community to quantify the robustness of DL models as well [2], [40], [41], [44].…”

“…Another popular line of work is deep learning testing, which aims to generate test cases that can expose the vulnerabilities of DL models. The test cases can then be used to improve the model robustness by retraining the model, however, this should not be taken as granted, as recent studies have shown that test cases generated based on existing testing metrics have limited correlation to model robustness and robustness improvement after retraining [6], [13]. In this work, we highlight and tackle the problem of effectively generating test cases for improving the adversarial robustness of DL models.…”

“…Existing test case generation techniques such as DeepXplore [30], DeepConcolic [33], DeepHunter [43] and ADAPT [21] are mostly designed to improve the neuron coverage metrics of the test cases. While existing testing approaches are helpful in exposing vulnerabilities of DL systems to some extent, recent studies have found that neuron coverage metrics are not useful for improving model robustness [6], [13], [23]. As a consequence, unlike in the case of traditional program testing (where the program is surely improved after fixing bugs revealed through testing), one may not improve the robustness of the DL system after testing.…”

Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source

“…Due to the popularity of DL models and the critical importance of their reliability, a growing body of research efforts have been dedicated to testing DL models, with focus on adversarial attacks [14,21,32,[46][47][48] for model robustness, the discussion on various metrics for DL model testing [36,39,43,52,69], and testing DL models for specific applications [63,71,78]. Meanwhile, both running and testing DL models inevitably involve the underlying DL libraries, which serve as central pieces of infrastructures for building, training, optimizing and deploying DL models.…”