Robust control of uncertain Markov Decision Processes with temporal logic specifications

Abstract

Abstract-We present a method for designing robust controllers for dynamical systems with linear temporal logic specifications. We abstract the original system by a finite Markov Decision Process (MDP) that has transition probabilities in a specified uncertainty set. A robust control policy for the MDP is generated that maximizes the worst-case probability of satisfying the specification over all transition probabilities in the uncertainty set. To do this, we use a procedure from probabilistic model checking to combine the system model with an automaton representing the specification. This new MDP is then transformed into an equivalent form that satisfies assumptions for stochastic shortest path dynamic programming. A robust version of dynamic programming allows us to solve for a -suboptimal robust control policy with time complexity O(log1 ) times that for the non-robust case. We then implement this control policy on the original dynamical system.

Strategy Synthesis for Autonomous Agents Using PRISM

“…There has been significant recent work using Markov decision processes, temporal logic specifications and model checking for generating controllers of autonomous systems. These works differ in the temporal logic specifications used and include approaches using the branching time logic PCTL [36,22], linear time temporal logic LTL [35,7], metric temporal logic [11], rewards [30] and multi-objective queries [23,21]. Also, both partially observable Markov decision processes, e.g.…”

Polynomial-Time Verification of PCTL Properties of MDPs with Convex Uncertainties

“…The proof follows closely the ones in [B.1] (Vol. II, Section 2.4) and [20]. Those proofs refer to a control setting, where the optimal action (control) can be selected.…”

“…Conversely, in the verification setting, the contraction needs to hold across all available actions, because we consider the worst case resolution of nondeterminism (universal quantification). Further, as in [20], we quantify across all nature behaviors: this is possible due to Assumption 2.2. For the sake of brevity, in the following we will only consider the calculation of P r min s , but the same reasoning applies also for the maximization problem.…”

“…The convex uncertainty models [11] analyzed in this paper have been considered recently in the robust control literature. In [20], an algorithm is given to synthesize a robust optimal controller for an MDP to satisfy a Linear Temporal Logic (LTL) specification where only one probabilistic operator is allowed. Their technique first converts the LTL specification to a Rabin automaton (which is worst-case doubly exponential in the size of the LTL formula), and composes it with the MDP.…”

Asymptotically Optimal Stochastic Motion Planning with Temporal Goals

“…These works are incomplete, however, in that methods to construct the approximating mdp for the robot are not presented; only planning over an existing abstraction is discussed. Uncertain mdps, where transition probabilities can belong to sets of values, have also been employed to provide a hierarchical abstraction and improve computational complexity [22]. Strong assumptions must be made on the structure of this abstraction, many of which are difficult to realize for physical systems.…”