Black-Box Testing of Deep Neural Networks through Test Case Diversity

Abstract

Deep Neural Networks (DNNs) have been extensively used in many areas including image processing, medical diagnostics and autonomous driving. However, DNNs can exhibit erroneous behaviours that may lead to critical errors, especially when used in safety-critical systems. Inspired by testing techniques for traditional software systems, researchers have proposed neuron coverage criteria, as an analogy to source code coverage, to guide the testing of DNNs. Despite very active research on DNN coverage, several recent studies have questioned the usefulness of such criteria in guiding DNN testing. Further, from a practical standpoint, these criteria are white-box as they require access to the internals or training data of DNNs, which is often not feasible or convenient. Measuring such coverage requires executing DNNs with candidate inputs to guide testing, which is not an option in many practical contexts. In this paper, we investigate diversity metrics as an alternative to white-box coverage criteria. For the previously mentioned reasons, we require such metrics to be black-box and not rely on the execution and outputs of DNNs under test. To this end, we first select and adapt three diversity metrics and study, in a controlled manner, their capacity to measure actual diversity in input sets. We then analyze their statistical association with fault detection using four datasets and five DNNs. We further compare diversity with state-of-the-art white-box coverage criteria. As a mechanism to enable such analysis, we also propose a novel way to estimate fault detection in DNNs.Our experiments show that relying on the diversity of image features embedded in test input sets is a more reliable indicator than coverage criteria to effectively guide DNN testing. Indeed, we found that one of our selected black-box diversity metrics far outperforms existing coverage criteria in terms of fault-revealing capability and computational time. Results also confirm the suspicions that state-of-the-art coverage criteria are not adequate to guide the construction of test input sets to detect as many faults as possible using natural inputs.

A lightweight adaptive random testing method for deep learning systems

“…This adaptive selection strategy for test cases has proven effective for numerical input spaces 13 . For the input space of DL systems, its applicability was also confirmed by some recent studies 20‐22 …”

“…13 For the input space of DL systems, its applicability was also confirmed by some recent studies. [20][21][22] Diversity, as a fundamental principle of test case selection, 14 can be expressed in various ways. Categorization is one such approach.…”

“…17,18 And comparative experiments have also demonstrated that ART has stronger failure detection capability than the basic random testing, often revealing failures in software systems with fewer test cases. 19 In order to expose the vulnerabilities in DL systems as many as possible, some studies 20,21 have shown that it is also necessary to construct or select diverse test inputs to execute deep neural network programs. Based on this point, Yan et al 22 applied the ART method to the testing of DL systems for image classification, and proposed the corresponding algorithm for DL systems, namely ARTDL.…”

Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles

“…Furthermore, the performance of most coverage measures is assessed using adversarial inputs, thus focussing on the robustness of the model instead of correctness. Irrespective of the claimed sensitivity of these measures to adversarial inputs, studies have failed to find a significant correlation between these coverage measures and their fault detection capability [5,18,48].…”

“…Another widely used adequacy measure for traditional and AIbased systems is test suite diversity computed on test inputs or outputs [5,13,29,50]. The diversity metrics are designed based on the intuition that similar test cases exercise similar parts of the source code or training examples, thus revealing the same faults.…”