Stochastic dynamic programming with factored representations

Abstract

Markov decision processes (MDPs) have proven to be popular models for decision-theoretic planning, but standard dynamic programming algorithms for solving MDPs rely on explicit, state-based specifications and computations. To alleviate the combinatorial problems associated with such methods, we propose new representational and computational techniques for MDPs that exploit certain types of problem structure. We use dynamic Bayesian networks (with decision trees representing the local families of conditional probability distributions) to represent stochastic actions in an MDP, together with a decision-tree representation of rewards. Based on this representation, we develop versions of standard dynamic programming algorithms that directly manipulate decision-tree representations of policies and value functions. This generally obviates the need for state-by-state computation, aggregating states at the leaves of these trees and requiring computations only for each aggregate state. The key to these algorithms is a decision-theoretic generalization of classic regression analysis, in which we determine the features relevant to predicting expected value. We demonstrate the method empirically on several planning problems,

Efficient Solution Algorithms for Factored MDPs

“…Our second evaluation compares a table-based implementation to a rule-based implementation that can exploit CSI. Finally, we present comparisons between our approach and the algorithms of Boutilier et al (2000).…”

“…This representation allows the algorithms to take advantage of both additive and context-specific structure, in order to produce high-quality approximate solutions very efficiently. The capability to exploit both types of structure distinguishes these algorithms differ from earlier approaches (Boutilier et al, 2000), which only exploit context-specific structure. We provide a more detailed discussion of the differences in Section 10.…”

“…Fortunately, as discussed by Koller and Parr (1999), this expectation operation, or backprojection, can be performed efficiently if the transition model and the value function are both factored appropriately. The linearity of the value function permits a linear decomposition, where each summand in the expectation can be viewed as an independent value function and updated in a manner similar to the value iteration procedure used by Boutilier et al (2000). We now recap the construction briefly, by first defining:…”

“…However, different actions often have very similar transition dynamics, only differing in their effect on some small set of variables. In particular, in many cases a variable has a default evolution model, which only changes if an action affects it directly (Boutilier et al, 2000).…”

“…However, many large MDPs have significant internal structure, and can be modeled compactly if the structure is exploited in the representation. (Boutilier, Dearden, & Goldszmidt, 2000) are one approach to representing large, structured MDPs compactly. In this framework, a state is implicitly described by an assignment to some set of state variables.…”