Counterexample-Guided Strategy Improvement for POMDPs Using Recurrent Neural Networks

Abstract

We study strategy synthesis for partially observable Markov decision processes (POMDPs). The particular problem is to determine strategies that provably adhere to (probabilistic) temporal logic constraints. This problem is computationally intractable and theoretically hard. We propose a novel method that combines techniques from machine learning and formal verification. First, we train a recurrent neural network (RNN) to encode POMDP strategies. The RNN accounts for memory-based decisions without the need to expand the full belief space of a POMDP. Secondly, we restrict the RNN-based strategy to represent a finite-memory strategy and implement it on a specific POMDP. For the resulting finite Markov chain, efficient formal verification techniques provide provable guarantees against temporal logic specifications. If the specification is not satisfied, counterexamples supply diagnostic information. We use this information to improve the strategy by iteratively training the RNN. Numerical experiments show that the proposed method elevates the state of the art in POMDP solving by up to three orders of magnitude in terms of solving times and model sizes.

Verifiable RNN-Based Policies for POMDPs Under Temporal Logic Constraints

“…Closest to the proposed method is [Carr et al, 2019], which introduced a verification-guided method to train RNNs as POMDP policies. In contrast to the proposed method, while polices are extracted from the RNNs, these policies do not directly exhibit the memory structure of the RNNs and are instead handcrafted based on knowledge about the particular application.…”

“…For that, we use diagnostic information in the form of counterexamples to generate new data [Carr et al, 2019].…”

“…Alternatively, one can generate data through bootstrapped random sampling as in [Hausknecht and Stone, 2015]. We follow the approach from [Carr et al, 2019] and start with a policy for the underlying MDP M that satisfies ϕ. This policy is gradually improved using a counterexample-guided technique employing the set Crit M Aγ .…”

“…For both sets we compare to two synthesis tools: PRISM-POMDP [Norman et al, 2017] and SolvePOMDP [Walraven and Spaan, 2017] from the respective formal methods and planning communities. We further compare to another RNN-based synthesis procedure with a handcrafted memory structure for FSCs [Carr et al, 2019]. For proper comparison with the synthesis tools we adopt the experiment setup of [Carr et al, 2019], whereby we always iterate for 10 instances of retraining the RNN from counterexample data.…”

“…We further compare to another RNN-based synthesis procedure with a handcrafted memory structure for FSCs [Carr et al, 2019]. For proper comparison with the synthesis tools we adopt the experiment setup of [Carr et al, 2019], whereby we always iterate for 10 instances of retraining the RNN from counterexample data. Similarly, the proposed approach is not guaranteed to reach the optimum, but shall rather improve as far as possible within 10 iterations.…”