Structural Test Coverage Criteria for Deep Neural Networks

Sun, Youcheng; Huang, Xiaowei; Kroening, Daniel; Sharp, James; Hill, Matthew Q.; Ashmore, Rob

doi:10.1145/3358233

Cited by 69 publications

(33 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We sample a number of tracks with maximum length of 30, and then apply the T est(s, k) with a variety of configurations: s ∈ [5,20], k ∈ [1,4]. Example tracks are as in Figure 3.…”

Section: Methodsmentioning

confidence: 99%

“…Several structural coverage criteria have been designed for DNNs, including neuron coverage [3] and its extensions [9], and variants of Multiple Condition/Decision Coverage (MC/DC) for DNNs [10]. These coverage criteria quantify the exhaustiveness of a test suite for a DNN.…”

Section: A the Deepconcolic Toolmentioning

confidence: 99%

“…A near-constant velocity model is applied as the dynamic model in the Kalman filter which is defined in (4).…”

Section: E Object Tracking 1) Problem Statementmentioning

confidence: 99%

“…Structural code coverage combined with requirements coverage has been the primary approach for measuring test completeness as assurance evidence to support safety arguments, for the certification of safety-critical software. Testing techniques for machine learning components, primarily DNNs, are comparatively new and have only been actively developed in the past few years, e.g., [3], [4]. Unlike model-based software systems, DNNs are usually considered as black boxes, and therefore it is difficult to understand their behaviour by means of inspection.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Reliability Validation of Learning Enabled Vehicle Tracking

Sun

Zhou

Maskell

et al. 2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

This paper studies the reliability of a real-world learning-enabled system, which conducts dynamic vehicle tracking based on a high-resolution wide-area motion imagery input. The system consists of multiple neural network componentsto process the imagery inputs-and multiple symbolic (Kalman filter) components-to analyse the processed information for vehicle tracking. It is known that neural networks suffer from adversarial examples, which make them lack robustness. However, it is unclear if and how the adversarial examples over learning components can affect the overall system-level reliability. By integrating a coverage-guided neural network testing tool, DeepConcolic, with the vehicle tracking system, we found that (1) the overall system can be resilient to some adversarial examples thanks to the existence of other components, and (2) the overall system presents an extra level of uncertainty which cannot be determined by analysing the deep learning components only. This research suggests the need for novel verification and validation methods for learning-enabled systems.

show abstract

“…We sample a number of tracks with maximum length of 30, and then apply the T est(s, k) with a variety of configurations: s ∈ [5,20], k ∈ [1,4]. Example tracks are as in Figure 3.…”

Section: Methodsmentioning

confidence: 99%

Section: A the Deepconcolic Toolmentioning

confidence: 99%

“…A near-constant velocity model is applied as the dynamic model in the Kalman filter which is defined in (4).…”

Section: E Object Tracking 1) Problem Statementmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Reliability Validation of Learning Enabled Vehicle Tracking

Sun

Zhou

Maskell

et al. 2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Their approach includes a test case generation algorithm that perturbs a given test case using linear programming with a goal to encode the test requirement and a fragment of the DNN. The same author also developed a test case generation algorithm based on symbolic approach and the gradient-based heuristic (Sun et al 2019). The difference between their coverage approach, based on MC/DC criterion, and neuron coverage is that the latter only considers individual activations of neurons, while the former considers causal relations between features at consecutive layers of the neural network.…”

Section: Test Coveragementioning

confidence: 99%

Software Testing for Machine Learning

Marijan

Gotlieb

2020

AAAI

View full text Add to dashboard Cite

Machine learning has become prevalent across a wide variety of applications. Unfortunately, machine learning has also shown to be susceptible to deception, leading to errors, and even fatal failures. This circumstance calls into question the widespread use of machine learning, especially in safety-critical applications, unless we are able to assure its correctness and trustworthiness properties. Software verification and testing are established technique for assuring such properties, for example by detecting errors. However, software testing challenges for machine learning are vast and profuse - yet critical to address. This summary talk discusses the current state-of-the-art of software testing for machine learning. More specifically, it discusses six key challenge areas for software testing of machine learning systems, examines current approaches to these challenges and highlights their limitations. The paper provides a research agenda with elaborated directions for making progress toward advancing the state-of-the-art on testing of machine learning.

show abstract

Revisiting deep neural network test coverage from the test effectiveness perspective

Yan

Chen

Cao

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

Many test coverage metrics have been proposed to measure the deep neural network (DNN) testing effectiveness, including structural coverage and nonstructural coverage. These test coverage metrics are proposed based on the fundamental assumption: They are correlated with test effectiveness. However, the fundamental assumption is still not validated sufficiently and reasonably, which brings question on the usefulness of DNN test coverage. This paper conducted a revisiting study on the existing DNN test coverage from the test effectiveness perspective, to effectively validate the fundamental assumption. Here, we carefully considered the diversity of subjects, three test effectiveness criteria, and both typical and state‐of‐the‐art test coverage metrics. Different from all the existing studies that deliver negative conclusions on the usefulness of existing DNN test coverage, we identified some positive conclusions on their usefulness from the test effectiveness perspective. In particular, we found the complementary relationship between structural and nonstructural coverage and identified the practical usage scenarios and promising research directions for these existing test coverage metrics.

show abstract

Structural Test Coverage Criteria for Deep Neural Networks

Cited by 69 publications

References 23 publications

Reliability Validation of Learning Enabled Vehicle Tracking

Reliability Validation of Learning Enabled Vehicle Tracking

Software Testing for Machine Learning

Revisiting deep neural network test coverage from the test effectiveness perspective

Contact Info

Product

Resources

About