Verification of Markov Decision Processes Using Learning Algorithms

Abstract

Abstract.We present a general framework for applying machine-learning algorithms to the verification of Markov decision processes (MDPs). The primary goal of these techniques is to improve performance by avoiding an exhaustive exploration of the state space. Our framework focuses on probabilistic reachability, which is a core property for verification, and is illustrated through two distinct instantiations. The first assumes that full knowledge of the MDP is available, and performs a heuristic-driven partial exploration of the model, yielding precise lower and upper bounds on the required probability. The second tackles the case where we may only sample the MDP, and yields probabilistic guarantees, again in terms of both the lower and upper bounds, which provides efficient stopping criteria for the approximation. The latter is the first extension of statistical model-checking for unbounded properties in MDPs. In contrast with other related approaches, we do not restrict our attention to time-bounded (finite-horizon) or discounted properties, nor assume any particular properties of the MDP. We also show how our techniques extend to LTL objectives. We present experimental results showing the performance of our framework on several examples.

Reachability in MDPs: Refining Convergence of Value Iteration

“…Moreover, we gave results over the convergence speed, as well as criteria for obtaining exact convergence. As future works, it seems particularly interesting to test this algorithm on real instances, as it is done in [2], where authors moreover apply machine learning techniques.…”

“…Interestingly, our approach has been realized in parallel of Brázdil et al [2] that solves a different problem with similar ideas. There, authors use some machine learning algorithm, namely real-time dynamic programming, in order to avoid to apply the full operator at each step of the value iteration, but rather to partially apply it based on some statistical test.…”

“…We thank the reviewer of RP 2014 that pointed out the similarities between our approach and [2] (to be presented at the next ATVA, in Nov. 2014).…”

Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle

“…Figure 7 illustrates the transition probability map at a velocity of 25 km/h. In this study, according to the Markov decision processes (MDPs) introduced in [26], the driving schedule was considered a finite MDP. The MDP comprises a set of state variables S = {(SOC(t), neng(t))| 0.2 ≤ SOC(t) ≤ 0.8, neng,min ≤ neng(t) ≤ neng,max}, set of actions a = {u_th(t)}, reward function r = f m  (s,a), and transition function psa, s', where psa, s' represents the probability of making a transition from state s to state s´ using action a.…”

Statistical Approximation of Optimal Schedulers for Probabilistic Timed Automata

“…For the former, some SMC-like approaches have recently been developed. They either work by iteratively optimising the decisions of an explicitly-stored scheduler [4,9], or by sampling from the scheduler space and iteratively improving a set of candidate near-optimal schedulers [5]. The former are heavyweight techniques because the size of the description of the (memoryless) scheduler is significant, and in the worst case is the size of the state space.…”