Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems

Abstract

Due to the increasing usage of machine learning (ML) techniques in security-and safety-critical domains, such as autonomous systems and medical diagnosis, ensuring correct behavior of ML systems, especially for different corner cases, is of growing importance. In this paper, we propose a generic framework for evaluating security and robustness of ML systems using different real-world safety properties. We further design, implement and evaluate VERIVIS, a scalable methodology that can verify a diverse set of safety properties for state-of-the-art computer vision systems with only blackbox access. VERIVIS leverage different input space reduction techniques for efficient verification of different safety properties. VERIVIS is able to find thousands of safety violations in fifteen state-of-the-art computer vision systems including ten Deep Neural Networks (DNNs) such as Inception-v3 and Nvidia's Dave self-driving system with thousands of neurons as well as five commercial third-party vision APIs including Google vision and Clarifai for twelve different safety properties. Furthermore, VERIVIS can successfully verify local safety properties, on average, for around 31.7% of the test images. VERIVIS finds up to 64.8× more violations than existing gradient-based methods that, unlike VERIVIS, cannot ensure non-existence of any violations. Finally, we show that retraining using the safety violations detected by VERIVIS can reduce the average number of violations up to 60.2%.

Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming

“…They report a significant performance drop of an object detection model when evaluated on corrupted data. Pei et al [2017] introduce VeriVis, a framework to evaluate the security and robustness of different object recognition models using real-world image corruptions such as brightness, contrast, rotations, smoothing, blurring, and others.…”

Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness

“…Robustness against simple transformations Approaches targeting adversarial accuracy for simple transformations have used attacks and defenses in the spirit of PGD (either on transformation space [11] or on input space projecting to transformation manifold [20]) and simple random or grid search [11,34]. Recent work [10] has also evaluated some rotation-equivariant networks with different training and attack settings which reduces direct comparability with e.g.…”

“…Recently, it was observed in [11,13,34,20,14,2] that worst-case prediction performance drops dramatically for neural network classifiers obtained using standard training, even for rather small transformation sets. In this context, we examine the effectiveness of regularization that explicitly encourages the predictor to be constant for transformed versions of the same image, which we refer to as being invariant on the transformation sets.…”

Taxonomy of Machine Learning Safety: A Survey and Primer

“…Additionally, as ML algorithms carry uncertainty in their outputs, the benefits of including prediction uncertainty in the overall system specification is an open and under investigation topic [151]. Focusing on invariance specifications, Pei et al [172] decompose safety properties for common real-world image distortions into 12 transformation invariance properties that a ML algorithm should maintain. Based on these specifications, they verify safety properties of the trained model using samples from the target domain.…”