Efficient Solution Algorithms for Factored MDPs

Abstract

This paper addresses the problem of planning under uncertainty in large Markov Decision Processes (MDPs). Factored MDPs represent a complex state space using state variables and the transition model using a dynamic Bayesian network. This representation often allows an exponential reduction in the representation size of structured MDPs, but the complexity of exact solution algorithms for such MDPs can grow exponentially in the representation size. In this paper, we present two approximate solution algorithms that exploit structure in factored MDPs. Both use an approximate value function represented as a linear combination of basis functions, where each basis function involves only a small subset of the domain variables. A key contribution of this paper is that it shows how the basic operations of both algorithms can be performed efficiently in closed form, by exploiting both additive and context-specific structure in a factored MDP. A central element of our algorithms is a novel linear program decomposition technique, analogous to variable elimination in Bayesian networks, which reduces an exponentially large LP to a provably equivalent, polynomial-sized one. One algorithm uses approximate linear programming, and the second approximate dynamic programming. Our dynamic programming algorithm is novel in that it uses an approximation based on max-norm, a technique that more directly minimizes the terms that appear in error bounds for approximate MDP algorithms. We provide experimental results on problems with over 10 40 states, demonstrating a promising indication of the scalability of our approach, and compare our algorithm to an existing state-of-the-art approach, showing, in some problems, exponential gains in computation time.

On Constraint Sampling in the Linear Programming Approach to Approximate Dynamic Programming

“…Morrison and Kumar (1999) formulate approximate linear programming algorithms for queueing problems with a specific choice of basis functions that renders all but a relatively small number of constraints redundant. Guestrin et al (2003) exploit the structure arising when factored linear architectures are used for approximating the cost-to-go function in factored MDPs. In some special cases, this allows for efficient exact solution of the ALP, and in others, this motivates alternative approximate solution methods.…”

“…Schuurmans and Patrascu (2002) devise a constraint generation scheme, also especially designed for factored MDPs with factored linear architectures. The worst-case computation time of this scheme grows exponentially with the number of state variables, even for special cases treated effectively by the methods of Guestrin et al (2003). However, the proposed scheme requires a smaller amount of computation time, on average.…”

Feature Article—Merging AI and OR to Solve High-Dimensional Stochastic Optimization Problems Using Approximate Dynamic Programming

“…Although the generality is attractive, this is precisely what makes general dynamic programs impossible to solve. It is far more effective to use a factored representation (Boutilier et al 1999, Guestrin et al 2003, where we retain the specific structure of the state variable. The term "flat representation" is also used in the AI community to distinguish nonhierarchical from hierarchical representations.…”

Analyse en norme Lp de l'algorithme d'itérations sur les valeurs avec approximations

“…De plus, elle n'est pas très utile en pratique puisque la plupart des opérateurs d'approximation et algorithmes d'AS résolvent un problème de minimisation en norme L 1 ou L 2 (bien que certains approximateurs de fonction utilisant la norme L ∞ , tels les averagers, aient été étudiés dans le cadre de la PD (Gordon, 1995, Guestrin et al, 2001). …”