Self-Checking Deep Neural Networks in Deployment

Xiao, Yan; Beschastnikh, Ivan; Rosenblum, David S.; Sun, Changsheng; Elbaum, Sebastian; Lin, Yun; Dong, Jin Song

doi:10.1109/icse43902.2021.00044

Cited by 27 publications

(36 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, their approach is only applicable to convex-margin based classifiers and not to NNs. Xiao et al (2021) suggest a self-checking mechanism for NN, where the features of the internal layers are used to check the reliability of predictions. In contrast, our approach uses predictive uncertainties obtained via a softmax function, which is rather simple to implement.…”

Section: Related Workmentioning

confidence: 99%

Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers

Andersen¹,

Maalej²

2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

To maximize the accuracy and increase the overall acceptance of text classifiers, we propose a framework for the efficient, inoperation moderation of classifiers' output. Our framework focuses on use cases in which F1-scores of modern Neural Networks classifiers (ca. 90%) are still inapplicable in practice. We suggest a semi-automated approach that uses prediction uncertainties to pass unconfident, probably incorrect classifications to human moderators. To minimize the workload, we limit the human moderated data to the point where the accuracy gains saturate and further human effort does not lead to substantial improvements. A series of benchmarking experiments based on three different datasets and three state-of-the-art classifiers show that our framework can improve the classification F1scores by 5.1 to 11.2% (up to approx. 98 to 99%), while reducing the moderation load up to 73.3% compared to a random moderation.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers

Andersen¹,

Maalej²

2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

show abstract

“…Their applicability to ImpNet is discussed in Section 6.1. Most are summaried by Li et al [44], and we also examine Xiao et al [45]'s runtime self-checking and Xiao et al [46]'s Metamorphic Testing.…”

Section: Defences Against ML Backdoorsmentioning

confidence: 99%

ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks

Clifford¹,

Shumailov²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Early backdoor attacks against machine learning set off an arms race in attack and defence development. Defences have since appeared demonstrating some ability to detect backdoors in models or even remove them. These defences work by inspecting the training data, the model, or the integrity of the training procedure. In this work, we show that backdoors can be added during compilation, circumventing any safeguards in the data preparation and model training stages. As an illustration, the attacker can insert weight-based backdoors during the hardware compilation step that will not be detected by any training or data-preparation process. Next, we demonstrate that some backdoors, such as ImpNet, can only be reliably detected at the stage where they are inserted and removing them anywhere else presents a significant challenge. We conclude that machine-learning model security requires assurance of provenance along the entire technical pipeline, including the data, model architecture, compiler, and hardware specification.

show abstract

“…However, they focus on adversarial attacks and there are relatively larger computation costs. Except for the testing on the trained model, some techniques (e.g., [50]) have been proposed to build the self-checking system that can monitor DNN output and trigger an alarm if the output is likely to be incorrect after the model is deployed. For more relevant discussions on the recent progress of machine learning testing, we refer the interesting readers to the recent comprehensive survey [54].…”

Section: Testingmentioning

confidence: 99%

NPC: Neuron Path Coverage via Characterizing Decision Logic of Deep Neural Networks

Xie¹,

Li²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning has recently been widely applied to many applications across different domains, e.g., image classification and audio recognition. However, the quality of Deep Neural Networks (DNNs) still raises concerns in the practical operational environment, which calls for systematic testing, especially in safetycritical scenarios. Inspired by software testing, a number of structural coverage criteria are designed and proposed to measure the test adequacy of DNNs. However, due to the blackbox nature of DNN, the existing structural coverage criteria are difficult to interpret, making it hard to understand the underlying principles of these criteria. The relationship between the structural coverage and the decision logic of DNNs is unknown. Moreover, recent studies have further revealed the non-existence of correlation between the structural coverage and DNN defect detection, which further posts concerns on what a suitable DNN testing criterion should be.In this paper, we propose the interpretable coverage criteria through constructing the decision structure of a DNN. Mirroring the control flow graph of the traditional program, we first extract a decision graph from a DNN based on its interpretation, where a path of the decision graph represents a decision logic of the DNN. Based on the control flow and data flow of the decision graph, we propose two variants of path coverage to measure the adequacy of the test cases in exercising the decision logic. The higher the path coverage, the more diverse decision logic the DNN is expected to be explored. Our large-scale evaluation results demonstrate that: the path in the decision graph is effective in characterizing the decision of the DNN, and the proposed coverage criteria are also sensitive with errors including natural errors and adversarial examples, and strongly correlated with the output impartiality.CCS Concepts: • Software and its engineering → Software testing and debugging; • Computing methodologies → Neural networks.

show abstract

Self-Checking Deep Neural Networks in Deployment

Cited by 27 publications

References 30 publications

Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers

Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers

ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks

NPC: Neuron Path Coverage via Characterizing Decision Logic of Deep Neural Networks

Contact Info

Product

Resources

About