The Complexity of Graph-Based Reductions for Reachability in Markov Decision Processes

Abstract

We study the never-worse relation (NWR) for Markov decision processes with an infinite-horizon reachability objective. A state q is never worse than a state p if the maximal probability of reaching the target set of states from p is at most the same value from q, regardless of the probabilities labelling the transitions. Extremal-probability states, end components, and essential states are all special cases of the equivalence relation induced by the NWR. Using the NWR, states in the same equivalence class can be collapsed. Then, actions leading to sub-optimal states can be removed. We show that the natural decision problem associated to computing the NWR is coNP-complete. Finally, we extend a previously known incomplete polynomial-time iterative algorithm to underapproximate the NWR.

Are Parametric Markov Chains Monotonic?

“…Pre-orders. A never-worse relation (NWR) on MDP states [6,43] is similar in spirit to reachability orders: states are ordered according to their maximal reachability probabilities but without taking the probabilities into account. Dependencies between the state probabilities are thus not taken into account.…”

“…NWR captures most heuristics to reduce the MDP before linear programming or value iteration. The coNP-completeness [43] indicates that checking this order is simpler than our pre-order unless coETR and coNP coincide.…”