Verifying Reinforcement Learning up to Infinity

Abstract

Formally verifying that reinforcement learning systems act safely is increasingly important, but existing methods only verify over finite time. This is of limited use for dynamical systems that run indefinitely. We introduce the first method for verifying the time-unbounded safety of neural networks controlling dynamical systems. We develop a novel abstract interpretation method which, by constructing adaptable template-based polyhedra using MILP and interval arithmetic, yields sound---safe and invariant---overapproximations of the reach set. This provides stronger safety guarantees than previous time-bounded methods and shows whether the agent has generalised beyond the length of its training episodes. Our method supports ReLU activation functions and systems with linear, piecewise linear and non-linear dynamics defined with polynomial and transcendental functions. We demonstrate its efficacy on a range of benchmark control problems.

Probabilistic Guarantees for Safe Deep Reinforcement Learning

“…(ii) Adaptive cruise control [2]: The problem has two vehicles i ∈ {lead , ego}, whose state is determined by variables x i and v i for the position and speed of each car, respectively. The lead car proceeds at constant speed (28 m s −1 ), and the agent controls the acceleration (±1 m s −2 ) of ego using two actions.…”

“…Formal verification of RL, but in a non-probabilistic setting includes: [5], which extracts and analyses decision trees; [27], which checks safety and liveness properties for deep RL; and [2], which also uses template polyhedra and MILP to build abstractions, but to check (non-probabilistic) safety invariants.…”

“…Lastly, we describe how to partition abstract states via refinement. We omit details of the environment abstraction since we reuse the symbolic post operator over template polyhedra given in [2], also performed with MILP. This supports environments specified as linear, piecewise linear or non-linear systems defined with polynomial and transcendental functions.…”

“…An alternative approach, which we take in this paper, is to verify an RL policy's correctness after it has been learnt, rather than placing restrictions on the learning process or on its deployment. Progress has been made in the formal verification of policies for RL [5] and also for the specific case of deep RL [27,2,3], in the latter case by building on advances in abstraction and verification techniques for neural networks; [2] also exploits the development of efficient abstract domains such as template polyhedra [41], previously applied to the verification of continuous-space and hybrid systems [6,15].…”

Verifying Learning-Based Robotic Navigation Systems

“…To date, the DNN verification community has focused primarily on feed-forward DNNs [21], [27], [30], [41], [68]. Some work has been carried out on verifying DRL networks, which pose greater challenges: beyond the general scalability challenges of DNNs verification, in DRL verification we must also take into account that agents typically interact with a reactive environment [5], [9], [13], [17]. In particular, these agents are invoked multiple times, and the inputs of each invocation are usually affected by the outputs of the previous invocations.…”