How to certify machine learning based safety-critical systems? A systematic literature review

Tambon, Florian; Laberge, Gabriel; An, Le; Nikanjam, Amin; Nouwou, Mindom, Paulina Stevia; Pequignot, Yann; Khomh, Foutse; Antoniol, G.; Merlo, Ettore; Laviolette, François

doi:10.1007/s10515-022-00337-x

Cited by 27 publications

(8 citation statements)

References 164 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In fact, they studied bugs in ML compilers (such as TVM 4 , Glow 5 , and nGraph 6 ), not in the ML-based systems. Tomban et al [97] also generated a list of bugs which may occur inside ML frameworks. Therefore, it is obvious that understanding about bugs in ML-based systems is still progressing.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Bugs in machine learning-based systems: a faultload benchmark

et al. 2023

View full text Add to dashboard Cite

Rapid growth of applying Machine Learning (ML) in different domains, especially in safety-critical areas, increases the need for reliable ML components, i.e., a software component operating based on ML. Since corrective maintenance, i.e. identifying and resolving systems bugs, is a key task in the software development process to deliver reliable software components, it is necessary to investigate the usage of ML components, from the software maintenance perspective. Understanding the bugs characteristics and maintenance challenges in ML-based systems can help developers of these systems to identify where to focus maintenance and testing efforts, by giving insights into the most error-prone components, most common bugs, etc. In this paper, we investigate the characteristics of bugs in ML-based software systems and the difference between ML and non-ML bugs from the maintenance viewpoint. We extracted 447,948 GitHub repositories that used one of the three most popular ML frameworks, i.e., TensorFlow, Keras, and PyTorch. After multiple filtering steps, we select the top 300 repositories with the highest number of closed issues. We manually investigate the extracted repositories to exclude non-MLbased systems. Our investigation involved a manual inspection of 386 sampled reported issues in the identified ML-based systems to indicate whether they affect ML components or not. Our analysis shows that nearly half of the real

show abstract

Section: Discussionmentioning

confidence: 99%

“…Some other studies [103,97,72] also investigated DL bugs, but inside the DL frameworks. For instance, Jia et al [72] studied root causes and symptoms of bugs affecting Tensorflow frameworks and provided a taxonomy.…”

Section: Related Workmentioning

confidence: 99%

Bugs in machine learning-based systems: a faultload benchmark

et al. 2023

View full text Add to dashboard Cite

show abstract

“…For each phase, the authors present example methods from the literature. Tambon et al (2022) present a systematic literature review covering 217 primary studies. The authors investigate fundamental topics such as robustness, uncertainty, explainability, and verification -and calls for deeper industry-academia collaborations.…”

Section: Related Workmentioning

confidence: 99%

“…Software systems developed for safety-critical applications must undergo assessments to demonstrate compliance with functional safety standards. However, as the conventional safety standards are not fully applicable for ML-enabled systems (Salay et al, 2018;Tambon et al, 2022), several domain-specific initiatives aim to complement them, e.g., organized by the EU Aviation Safety Agency, the ITU-WHO Focus Group on AI for Health, and the International Organization for Standardization.…”

Section: Introductionmentioning

confidence: 99%

Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System

Borg¹,

Henriksson²,

Socha³

et al. 2022

Preprint

View full text Add to dashboard Cite

Integration of Machine Learning (ML) components in critical applications introduces novel challenges for software certification and verification. New safety standards and technical guidelines are under development to support the safety of ML-based systems, e.g., ISO 21448 SOTIF for the automotive domain and the Assurance of Machine Learning for use in Autonomous Systems (AMLAS) framework. SOTIF and AMLAS provide high-level guidance but the details must be chiseled out for each specific case. We report results from an industry-academia collaboration on safety assurance of SMIRK, an ML-based pedestrian

show abstract

“…However, DNNs may produce unexpected or incorrect results that could lead to significant negative consequences or losses. Therefore, effective testing of such models is crucial to ensure their reliability [4]. For testing and enhancing the performance of DNN-driven applications, a significant amount of labeled data is required.…”

Section: Introductionmentioning

confidence: 99%

DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks

Aghababaeyan¹,

Abdellatif²,

Dadkhah³

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) are widely used in various application domains such as image processing, speech recognition, and natural language processing. However, testing DNN models may be challenging due to the complexity and size of their input domain. Particularly, testing DNN models often requires generating or exploring large unlabeled datasets. In practice, DNN test oracles, which identify the correct outputs for inputs, often require expensive manual effort to label test data, possibly involving multiple experts to ensure labeling correctness.In this paper, we propose DeepGD, a black-box multi-objective test selection approach for DNN models. It reduces the cost of labeling by prioritizing the selection of test inputs with high faultrevealing power from large unlabeled datasets. DeepGD not only selects test inputs with high uncertainty scores to trigger as many mispredicted inputs as possible but also maximizes the probability of revealing distinct faults in the DNN model by selecting diverse mispredicted inputs.The experimental results conducted on four widely used datasets and five DNN models show that in terms of fault-revealing ability:(1) White-box, coverage-based approaches fare poorly, (2) DeepGD outperforms existing black-box test selection approaches in terms of fault detection, and (3) DeepGD also leads to better guidance for DNN model retraining when using selected inputs to augment the training set.

show abstract

How to certify machine learning based safety-critical systems? A systematic literature review

Cited by 27 publications

References 164 publications

Bugs in machine learning-based systems: a faultload benchmark

Bugs in machine learning-based systems: a faultload benchmark

Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System

DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks

Contact Info

Product

Resources

About