Robust Control of Markov Decision Processes with Uncertain Transition Matrices

Abstract

Optimal solutions to Markov decision problems may be very sensitive with respect to the state transition probabilities. In many practical problems, the estimation of these probabilities is far from accurate. Hence, estimation errors are limiting factors in applying Markov decision processes to real-world problems.We consider a robust control problem for a finite-state, finite-action Markov decision process, where uncertainty on the transition matrices is described in terms of possibly nonconvex sets. We show that perfect duality holds for this problem, and that as a consequence, it can be solved with a variant of the classical dynamic programming algorithm, the "robust dynamic programming" algorithm. We show that a particular choice of the uncertainty sets, involving likelihood regions or entropy bounds, leads to both a statistically accurate representation of uncertainty, and a complexity of the robust recursion that is almost the same as that of the classical recursion. Hence, robustness can be added at practically no extra computing cost. We derive similar results for other uncertainty sets, including one with a finite number of possible values for the transition matrices.We describe in a practical path planning example the benefits of using a robust strategy instead of the classical optimal strategy; even if the uncertainty level is only crudely guessed, the robust strategy yields a much better worst-case expected travel time. Notation P > 0 or P 0 refers to the strict or nonstrict componentwise inequality for matrices or vectors. For a vector p > 0, log p refers to the componentwise operation. The notation 1 refers to the vector of ones, with size determined from context. The probability simplex in R n is denoted n = p ∈ R n + p T 1 = 1 , while n is the set of n × n transition matrices (componentwise nonnegative matrices with rows summing to one). We use to denote the support function of a set ⊆ R n , with for v ∈ R n , v = sup p T v p ∈ .

Robust Dynamic Programming

“…In Section 4 we describe three families of sets of conditional measures that are based on the confidence regions, and show that the computational effort required to solve the robust DP corresponding to these sets is only modestly higher than that required to solve the nonrobust counterpart. The results in this section, although independently obtained, are not new and were first obtained by Nilim and El Ghaoui (2002). In Section 5 we provide basic examples and computational results.…”

“…In current practice, these errors are ignored and the optimal policy is computed assuming that the estimate is, indeed, the true transition probability. The DP optimal policy is quite sensitive to perturbations in the transition probability and ignoring the estimation errors can lead to serious degradation in performance (Nilim and El Ghaoui, 2002;Tsitsiklis et al, 2002). Degradation in performance due to estimation errors in parameters has also been observed in other contexts (Ben-Tal and Nemirovski, 1997;Goldfarb and Iyengar, 2003).…”

“…While this paper was being prepared for publication we became aware of a technical report by Nilim and El Ghaoui (2002) where they formulate finite horizon robust DP in the context of an aircraft routing problem. A "robust counterpart" for the Bellman equation appears in their paper but they do not justify that this "robust counterpart", indeed, characterizes the robust value function.…”

Robust optimal policies for Markov decision processes with safety-threshold constraints

“…The robust optimal policy for an uncertain MDP can be computed using various methods [10], [16], [15].…”

“…However, none of these methods is robust in the presence of modeling uncertainty. On the other hand, for MDPs with uncertain parameters, robust MDPs have been extensively studied [10], [8], [15]. Recently, robust control of MDPs has been extended to handle expressive temporal logic constraints [16], [3], [14].…”