Unifying Evaluation of Machine Learning Safety Monitors

Guérin, Joris; Ferreira, Raul Sena; Delmas, Kévin; Guiochet, Jérémie

doi:10.1109/issre55969.2022.00047

Cited by 4 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a next step, we intend to add a scenario generation process similar to Abdessalem et al [38], capable of varying the scenarios, their objects and parameters. Besides, new safety-oriented metrics [39] can also be added to SiMOOD, which can produce more types of analysis.…”

Section: Discussionmentioning

confidence: 99%

SiMOOD: Evolutionary Testing Simulation With Out-Of-Distribution Images

Ferreira

Guérin

Guiochet

et al. 2022

2022 IEEE 27th Pacific Rim International Symposium on Dependable Computing (PRDC)

Self Cite

View full text Add to dashboard Cite

Testing perception functions for safety-critical autonomous systems is a crucial task. The reason is that accurate ML models applied in computer vision tasks still fail in scenarios where humans perform well. Out-of-distribution (OOD) images are usually a source of such failures. For this reason, literature usually applies data augmentation techniques or runtime monitors such as OOD detectors to increase robustness. Evaluating such solutions is usually performed by analyzing metrics based on positive and negative rates over a dataset containing several perturbations. However, using such metrics on such datasets can be misleading since not all OOD data lead to failures in the perception system. Hence, testing a perception system cannot be reduced to measuring Machine Learning (ML) performances on a dataset but rely on the images captured by the system at runtime. However, the amount of time spent to generate diverse test cases during a simulation of perception components can grow quickly since it is a combinatorial optimization problem. Aiming to provide a solution for this challenging task, we present SiMOOD, an evolutionary simulation testing of safety-critical perception systems, which comes integrated into the CARLA simulator. Unlike related works that simulate scenarios that raise failures for control or specific perception problems such as adversarial and novelty, we provide an approach that finds the most relevant OOD perturbations that can lead to hazards in safety-critical perception systems. Moreover, our approach can decrease, at least 10 times, the amount of time to find a set of hazards in safety-critical scenarios such as autonomous emergency braking system simulation. Besides, code is publicly available for use.

show abstract

Section: Discussionmentioning

confidence: 99%

SiMOOD: Evolutionary Testing Simulation With Out-Of-Distribution Images

Ferreira

Guérin

Guiochet

et al. 2022

2022 IEEE 27th Pacific Rim International Symposium on Dependable Computing (PRDC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…It would be beneficial to change the failure semantics of classifiers from uncontrolled content failures (i.e., misclassifications) to omission failures. Fail-controlled components [40] wrappers or monitors [13], [17], [30]. Safety wrappers are intended to complement an existing critical component or task by continuously checking invariants, or processing additional data to detect dangerous behaviors and block the erroneous output of the component before it is propagated through the system.…”

Section: Failure Modes Of Classifiers and Safety Wrappersmentioning

confidence: 99%

“…Figure 1b still feeds the input data to the classifier, which predicts the class dp_label for an input data dp. However, the adoption of a safety wrapper SW(clf) provides the input data and the class prediction of the clf to a misclassification detector, which outputs a binary confidence score [30] (BCS) to decide if the class prediction is detected to be a misclassification. In this case, the wrapper omits the output (with probability φ); otherwise, the class prediction gets forwarded to the encompassing system, is correct with probability α w and is a misclassification with probability w .…”

Section: Safety Wrappers For Black-box Classifiersmentioning

confidence: 99%

“…A typical approach to guarantee safety [40] is to equip a functional component with a detector [30], [33], [36], [37] so to trigger a failsafe or fail-stop behavior whenever the correct functioning is not guaranteed. At the system level, it is often desirable that safetycritical functions would either i) deliver a correct result or ii) omit outputs i.e., the function should have fail-omission failures only.…”

Section: Introductionmentioning

confidence: 99%

“…This is a well-known limitation to the adoption of classifiers to operate safety-critical functions, requiring countermeasures that avoid or mitigate the potential cascading effects of misclassifications to the encompassing system. A fail-omission classifier would either produce trusted outputs or omit them [30]. Clearly, this approach is different from building a classifier that never outputs misclassifications, which is unrealistic to assess at the state of the art due to the dimension of the input space and the unpredictable behavior with inputs close to the decision boundaries [34].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Ensembling Uncertainty Measures to Improve Safety of Black-Box Classifiers

Zoppi,

Ceccarelli,

Bondavalli

2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

Machine Learning (ML) algorithms that perform classification may predict the wrong class, experiencing misclassifications. It is well-known that misclassifications may have cascading effects on the encompassing system, possibly resulting in critical failures. This paper proposes SPROUT, a Safety wraPper thROugh ensembles of UncertainTy measures, which suspects misclassifications by computing uncertainty measures on the inputs and outputs of a black-box classifier. If a misclassification is detected, SPROUT blocks the propagation of the output of the classifier to the encompassing system. The resulting impact on safety is that SPROUT transforms erratic outputs (misclassifications) into data omission failures, which can be easily managed at the system level. SPROUT has a broad range of applications as it fits binary and multi-class classification, comprising image and tabular datasets. We experimentally show that SPROUT always identifies a huge fraction of the misclassifications of supervised classifiers, and it is able to detect all misclassifications in specific cases. SPROUT implementation contains pre-trained wrappers, it is publicly available and ready to be deployed with minimal effort.

show abstract

Characterizing Reliability of Three-version Traffic Sign Classifier System through Diversity Metrics

Wen,

Machida

2023

2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)

View full text Add to dashboard Cite

Unifying Evaluation of Machine Learning Safety Monitors

Cited by 4 publications

References 21 publications

SiMOOD: Evolutionary Testing Simulation With Out-Of-Distribution Images

SiMOOD: Evolutionary Testing Simulation With Out-Of-Distribution Images

Ensembling Uncertainty Measures to Improve Safety of Black-Box Classifiers

Characterizing Reliability of Three-version Traffic Sign Classifier System through Diversity Metrics

Contact Info

Product

Resources

About