On the Robustness of a Neural Network

Abstract

With the development of neural networks based machine learning and their usage in mission critical applications, voices are rising against the black box aspect of neural networks as it becomes crucial to understand their limits and capabilities. With the rise of neuromorphic hardware, it is even more critical to understand how a neural network, as a distributed system, tolerates the failures of its computing nodes, neurons, and its communication channels, synapses. Experimentally assessing the robustness of neural networks involves the quixotic venture of testing all the possible failures, on all the possible inputs, which ultimately hits a combinatorial explosion for the first, and the impossibility to gather all the possible inputs for the second.In this paper, we prove an upper bound on the expected error of the output when a subset of neurons crashes. This bound involves dependencies on the network parameters that can be seen as being too pessimistic in the average case. It involves a polynomial dependency on the Lipschitz coefficient of the neurons' activation function, and an exponential dependency on the depth of the layer where a failure occurs. We back up our theoretical results with experiments illustrating the extent to which our prediction matches the dependencies between the network parameters and robustness. Our results show that the robustness of neural networks to the average crash can be estimated without the need to neither test the network on all failure configurations, nor access the training set used to train the network, both of which are practically impossible requirements.

Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry

“…Their tool demonstrated an end-to-end driving application where the input is images and the output is the steering angle. Mhamdi et al also studied the black box aspects of neural networks, and show that the robustness of a complete DNN can be assessed by an analysis focused on individual neurons as units of failure 62 -a much more reasonable approach given the state-space explosion.…”

ResiliNet: Failure-Resilient Inference in Distributed Neural Networks

“…Neural Network Robustness. A line of research related to our study is robust neural networks (Goodfellow et al, 2015;Szegedy et al, 2014;Cisse et al, 2017;Bas-tani et al, 2016;El Mhamdi et al, 2017). Robustness in neural networks has gained considerable attention lately, and is especially important when the neural network are to be developed in commercial products.…”

Exploratory Data Analysis and Artificial Neural Network for Prediction of Leptospirosis Occurrence in Seremban, Malaysia Based on Meteorological Data

“…The range of differences are 0.0004 until 0.0006. Building robust ANN models never comes without a cost (Mhamdi et al, 2017). It is clearly shown in Figure 6 that the model for Seremban City is robust as it recorded the second smallest variance, which is 6.6379e-05, even though the prediction accuracy is not as high as the model from Perentian, which has the highest AUC value.…”