Formal Analysis of AI-Based Autonomy: From Modeling to Runtime Assurance

Abstract

Autonomous systems are increasingly deployed in safetycritical applications and rely more on high-performance AI/ML-based components. Runtime monitors play an important role in raising the level of assurance in AI/ML-based autonomous systems by ensuring that the autonomous system stays safe within its operating environment. In this tutorial, we present VerifAI, an open-source toolkit for the formal design and analysis of systems that include AI/ML components. Ver-ifAI provides features supporting a variety of use cases including formal modeling of the autonomous system and its environment, automatic falsification of system-level specifications as well as other simulationbased verification and testing methods, automated diagnosis of errors, and automatic specification-driven parameter and component synthesis. In particular, we describe the use of VerifAI for generating runtime monitors that capture the safe operational environment of systems with AI/ML components. We illustrate the advantages and applicability of VerifAI in real-life applications using a case study from the domain of autonomous aviation.

Learning Monitorable Operational Design Domains for Assured Autonomy

“…We show that our counterexample-guided approach can be used to learn a monitorable ODD for an image-based neural network used for lane keeping. Our example is inspired by several real case studies VerifAI has been applied in including with industrial partners (e.g., see [18,21,46]).…”