ARTDL: Adaptive Random Testing for Deep Learning Systems

Abstract

With recent breakthroughs in Deep Learning (DL), DL systems are increasingly deployed in safety-critical fields. Hence, some software testing methods are required to ensure the reliability and safety of DL systems. Since the rules of DL systems are inferred from training data, it is difficult to know the implementation rules about each behavior of DL systems. At the same time, Random Testing (RT) is a popular testing method and the knowledge about software implementation is not needed when we use RT. Therefore, RT is very suitable for the testing of DL systems. And the existing mechanisms for testing DL systems also depend heavily on RT by the labeled test data. In order to increase the effectiveness of RT for DL systems, we design, implement and evaluate the Adaptive Random Testing for DL systems (ARTDL), which is the first Adaptive Random Testing (ART) method to improve the effectiveness of RT for DL systems. ARTDL refers to the idea of ART. That is, fewer test cases are needed to detect failures by selecting the test case with the furthest distance from non-failure-causing test cases. Firstly, we propose the Feature-based Euclidean Distance (FED) as the distance metric that can be used to measure the difference between failure-causing inputs and non-failure-causing inputs. Secondly, we verify the availability of FED by presenting the failure pattern of DL models. Finally, we design ARTDL algorithm to generate the test cases that are more likely to cause failures based on the FED. We implement ARTDL to test top performing DL models in the field of image classification and automatic driving. The results show that, on average, the number of test cases used to find the first bug is reduced by 62.74% through ARTDL, compared with RT. INDEX TERMS Deep learning testing, adaptive random testing, distance metric, metamorphic testing.

How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review

“…If coverage based generation is the most widespread technique used, we identified techniques utilizing other methods to manage test cases generation for testing purposes. On image datasets, techniques based on Adaptive Random test [243] leveraging PCA decomposition of network's features, Lowdiscrepancy among sequences of images and Active Learning [53], with metamorphic testing either using entropy based technique with softmax predictions [218] or through the search of critical images following those transformations [169]. Note that metamorphic testing [34] is an effective proxy for image generation and testing which is also used by techniques mentioned previously such as DeepTest.…”

A Survey on Deep Learning for Steering Angle Prediction in Autonomous Vehicles

“…For example, [83] hybridized CNN and long-short term memory (LSTM)/recurrent neural network (RNN) to predict the steering angle. Three DNNbased steering angle prediction algorithms were independently designed by Chauffeur [84], Autumn [84], and Rambo [85]. The Chauffeur model included one CNN for extracting features from image and LSTM/RNN model for steering angle prediction; the Autumn model had five (5) CNNs layers connected together and an LSTM/RNN layer, and the Rambo model had three (3) CNNs layers whose output was combined at the final layer.…”

Return on Investment in Machine Learning: Crossing the Chasm between Academia and Business

“…They discuss testing of the data itself, but also testing ML infrastructure, model development, and methods of ML monitoring. There are also methodologies developed for particular branches of ML, such as methodologies dedicated to deep learning testing [47,56,114], testing for machine translation [102], or computing bounds on ML assurances from manifold learning [45]. A comprehensive survey of over 140 scientific papers on ML testing is available [118].…”