Towards Dependability Metrics for Neural Networks

Cheng, Chih-Hong; Nührenberg, Georg; Huang, Chung-Hao; Rueß, Harald; Yasuoka, Hirotoshi

doi:10.1109/memcod.2018.8556962

Cited by 34 publications

(24 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bugs in data affect the quality of the generated model, and can be amplified to yield more serious problems over a period a time [45]. Bug detection in data checks problems such as whether the data is sufficient for training or test a model (also called completeness of the data [46]), whether the data is representative of future data, whether the data contains a lot of noise such as biased labels, whether there is skew between training data and test data [45], and whether there is data poisoning [47] or adversary information that may affect the model's performance. Bug Detection in Frameworks.…”

Section: Testing Componentsmentioning

confidence: 99%

“…Automatic Assessment of Interpretability. Cheng et al [46] presented a metric to understand the behaviours of an ML model. The metric measures whether the learned has actually learned the object in object identification scenario via occluding the surroundings of the objects.…”

Section: Interpretabilitymentioning

confidence: 99%

“…Cheng et al [46] presented a series of metrics to evaluate whether the training data have covered all important scenarios. Performance-based Data Bug Detection.…”

Section: Bug Detection In Training Datamentioning

confidence: 99%

See 2 more Smart Citations

Machine Learning Testing: Survey, Landscapes and Horizons

Zhang

Harman

Ма

et al. 2022

IIEEE Trans. Software Eng.

501

318

View full text Add to dashboard Cite

This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 128 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing. Index Terms-machine learning, software testing, deep neural network, ! • Jie M. Zhang and Mark Harman are with CREST, University College London, United Kingdom. Mark Harman is also with Facebook London.

show abstract

Section: Testing Componentsmentioning

confidence: 99%

Section: Interpretabilitymentioning

confidence: 99%

See 1 more Smart Citation

Machine Learning Testing: Survey, Landscapes and Horizons

Zhang

Harman

Ма

et al. 2022

IIEEE Trans. Software Eng.

501

318

View full text Add to dashboard Cite

show abstract

“…Reluplex is an SMT (satisfiability modulo theories) solver [19] to verify properties of deep NNs or provide counterexamples against them by utilizing the simplex method [18] and the partial linearity of the ReLU function. Dependability metrics set for NNs is a related work that proposes metrics such as scenario coverage, neuron activation pattern, interpretation precision for RICC (robustness, interpretability, completeness, and correctness) criteria [13].…”

Section: Related Work and Research Directions For Verification Of Mamentioning

confidence: 99%

Engineering problems in machine learning systems

Kuwajima

Yasuoka²,

Nakae³

2020

Mach Learn

Self Cite

View full text Add to dashboard Cite

Fatal accidents are a major issue hindering the wide acceptance of safety-critical systems that employ machine learning and deep learning models, such as automated driving vehicles. In order to use machine learning in a safety-critical system, it is necessary to demonstrate the safety and security of the system through engineering processes. However, thus far, no such widely accepted engineering concepts or frameworks have been established for these systems. The key to using a machine learning model in a deductively engineered system is decomposing the data-driven training of machine learning models into requirement, design, and verification, particularly for machine learning models used in safety-critical systems. Simultaneously, open problems and relevant technical fields are not organized in a manner that enables researchers to select a theme and work on it. In this study, we identify, classify, and explore the open problems in engineering (safety-critical) machine learning systems -that is, in terms of requirement, design, and verification of machine learning models and systems -as well as discuss related works and research directions, using automated driving vehicles as an example. Our results show that machine learning models are characterized by a lack of requirements specification, lack of design specification, lack of interpretability, and lack of robustness. We also perform a gap analysis on a conventional system quality standard SQuARE with the characteristics of machine learning models to study quality models for machine learning systems. We find that a lack of requirements specification and lack of robustness have the greatest impact on conventional quality models.Preprint. Work in progress.

show abstract

“…Figure 1 provides a simplified Goal Structuring Notation (GSN) [18] diagram to assist in understanding how features provided by nn-dependability-kit contribute to the overall safety goal 1 . Our proposed metrics, unless explicitly specified, are based on extensions of our early work [5]. Starting with the goal of having a neural network to function correctly (G1), based on assumptions where no software and hardware fault appears (A1, A2), the strategy (S1) is to ensure that within different phases of the product life cycle, correctness is ensured.…”

Section: Introductionmentioning

confidence: 99%

nn-dependability-kit: Engineering Neural Networks for Safety-Critical Autonomous Driving Systems

Cheng

Huang

Nührenberg

2019

2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)

Self Cite

View full text Add to dashboard Cite

Can engineering neural networks be approached in a disciplined way similar to how engineers build software for civil aircraft? We present nn-dependability-kit, an opensource toolbox to support safety engineering of neural networks for autonomous driving systems. The rationale behind nndependability-kit is to consider a structured approach (via Goal Structuring Notation) to argue the quality of neural networks. In particular, the tool realizes recent scientific results including (a) novel dependability metrics for indicating sufficient elimination of uncertainties in the product life cycle, (b) formal reasoning engine for ensuring that the generalization does not lead to undesired behaviors, and (c) runtime monitoring for reasoning whether a decision of a neural network in operation is supported by prior similarities in the training data. A proprietary version of nndependability-kit has been used to improve the quality of a level-3 autonomous driving component developed by Audi for highway maneuvers.

show abstract

Towards Dependability Metrics for Neural Networks

Cited by 34 publications

References 13 publications

Machine Learning Testing: Survey, Landscapes and Horizons

Machine Learning Testing: Survey, Landscapes and Horizons

Engineering problems in machine learning systems

nn-dependability-kit: Engineering Neural Networks for Safety-Critical Autonomous Driving Systems

Contact Info

Product

Resources

About