Counterexample-guided permissive supervisor synthesis for probabilistic systems through learning

Abstract

Formal methods in robotic motion planning have emerged as a hot research topic recently due to its correctby-design nature, and most results haven been based on nonprobabilistic discrete models. To better handle the environment uncertainties, sensor noise and actuator imperfection, control problems in probabilistic systems like Markov Chain (MC) and Markov Decision Process (MDP) have also been studied. Most existing methods are either based on probabilistic model checking or through reinforcement learning oriented optimization. On the other hand, in the literature of supervisory control of discrete event systems, people usually design supervisors with maximum permissive nature. In other words, a collection of schedulers, instead of a single one scheduler, that satisfy the given specification is designed at the same time. We are therefore motivated to propose a novel learning based automated supervisor synthesis framework to automatically generate permissive supervisor so that the supervised system satisfies the given specification. Our approach is based on a modified L* learning algorithm and runs iteratively. It is guaranteed to be correct and terminate in finite steps.

Permissive Supervisor Synthesis for Markov Decision Processes Through Learning

“…This paper merges and further develops our previous results [22], [23]. While the same L* learning algorithm [14] was considered in synthesis process, we changed the form of the supervisor and considered positive counterexamples so that the results can become less conservative.…”

“…The resulting π i will then be passed to the third stage to answer the conjecture in our modified L* learning algorithm. Note that in our previous work [22], [23], we proposed to apply the hop-constrained k shortest path (HKSP) algorithm to find the smallest k paths so that eliminating them will make sure the DTMC will not be counterexample any more. The complexity of HKSP is higher than the HSP since HKSP has to search for k as well.…”

“…We assume that any a i ∈ A i , ∀i ∈ [1, N ] is controllable meaning that we could disable or enable it if we want. Note that in our previous results [22], [23], the local supervisor's alphabet Σ i = A i . From the definition of the parallel composition ||, it can be seen that it only keeps track of the action sequence but not state sequence.…”

Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamples

“…The L* learning approach assumes the existence of a teacher who can answer the membership and equivalence queries (Angluin 1987;Wu and Lin 2015;Wu, Zhang, and Lin 2018;Zhang, Wu, and Lin 2015;. Our approach fulfills the role of the teacher by an RL engine and the queries are answered through interaction with the environment through the RL engine.…”

Supervisor Synthesis of POMDP based on Automata Learning

“…Also due to the partial observability in POMDPs, applying these methods to POMDP are fundamentally difficult. Besides these works, using the L * algorithm to learn system supervisor has also been considered in our previous work for MDPs [42]. To apply the L * algorithm to POMDP supervisor synthesis, in this paper, we extensively discuss the supervisor synthesis framework and design new membership query and conjecture checking rules to overcome the difficulties brought by partial observability.…”