A Review of Formal Methods applied to Machine Learning

Abstract

We review state-of-the-art formal methods applied to the emerging field of the verification of machine learning systems. Formal methods can provide rigorous correctness guarantees on hardware and software systems. Thanks to the availability of mature tools, their use is well established in the industry, and in particular to check safety-critical applications as they undergo a stringent certification process. As machine learning is becoming more popular, machine-learned components are now considered for inclusion in critical systems. This raises the question of their safety and their verification. Yet, established formal methods are limited to classic, i.e. non machine-learned software. Applying formal methods to verify systems that include machine learning has only been considered recently and poses novel challenges in soundness, precision, and scalability.We first recall established formal methods and their current use in an exemplar safety-critical field, avionic software, with a focus on abstract interpretation based techniques as they provide a high level of scalability. This provides a golden standard and sets high expectations for machine learning verification. We then provide a comprehensive and detailed review of the formal methods developed so far for machine learning, highlighting their strengths and limitations. The large majority of them verify trained neural networks and employ either SMT, optimization, or abstract interpretation techniques. We also discuss methods for support vector machines and decision tree ensembles, as well as methods targeting training and data preparation, which are critical but often neglected aspects of machine learning. Finally, we offer perspectives for future research directions towards the formal verification of machine learning systems.

The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

“…It is possible to apply formal methods (mathematical techniques of verification) to ML models including for assessing properties such as robustness. A recent survey of such approaches can be found in [54]; it covers static analysis as well as methods such as Satisfiability Module Theories (SMT) and identifies some tools that are capable of scaling to very large models, e.g. NNs with millions of neurons.…”

An Empirical Analysis of the Use of Real-Time Reachability for the Safety Assurance of Autonomous Vehicles

“…Moreover, these systems often leverage machine learning components to deal with the diverse data obtained from the system's sensors. Applying formal assurance techniques to machine learning systems has only been considered recently and poses unique challenges [98]. While there has been significant progress within this realm, there is still a significant gap between the machine learning models that these approaches can handle, and the models deployed in state-of-the-art systems.…”

Trustworthy AI: From Principles to Practices

“…To solve this problem, the certified robustness approaches, such as CNN-Cert [59], CROWN [432], Fast-lin, or Fast-lip [396], etc., aim to minimize an upper bound of the worst-case loss. A recent survey work by [376] has a thorough review on formal verification for the neural network. They mainly categorize the formal methods into complete verification and incomplete verification methods.…”