Using Randomization to Break the Curse of Dimensionality

Abstract

USING RANDOMIZATION TO BREAK THE CURSE OF DIMENSIONALITY BY JOHN RUST' This paper introduces random versions of successive approximations and multigrid algorithms for computing approximate solutions to a class of finite and infinite horizon Markovian decision problems (MDPs). We prove that these algorithms succeed in breaking the "curse of dimensionality" for a subclass of MDPs known as discrete decision processes (DDPs).

Untitled

“…Similarly, in infinite-horizon problems we compute a solution toĴ = Tˆ aĴ by using the infinite-horizon version of value iteration. It is helpful to note thatˆ a is "self-approximating" in the sense that in order to characterizeĴ it suffices to find a set of function values at the locations y a s satisfyingĴ (y a s ) = (Tˆ aĴ )(y a s ) (see also Rust, 1997). Then the value ofˆ a J (x) at new locations x = y a s can be derived directly from the definition ofˆ a in (3).…”

“…The proof of Lemma 1, which is analogous to Rust's proof of a corresponding theorem regarding density-based random operators (Rust, 1997), can be found in Appendix A.2. Note that we restrict ourselves to the interval [b, 1 − b] to avoid boundary effects of the weighting kernel.…”

“…Besides guiding the practical choice of b, Theorem 3 provides insight into the question of whether simulation-based methods are suited to "break" the curse of dimensionality. This question is the main focus of Rust's analysis of density-based random approximations of the Bellman operator (Rust, 1997). In particular, Rust concludes that if the transition density p(y, | x, a) is known, an algorithm similar to ours can be used to approximate the value function in a computation time that is only polynomial in d. By contrast, Theorem 3 suggests that for kernel-based reinforcement learning, where p(y | x, a) is unknown, the number of observations grows exponentially in d even if b is chosen optimally.…”

“…In the context of reinforcement learning, local averaging has been suggested in work by Rust (1997) and Gordon (1999), making the assumption that the transition probabilities of the MDP are known and can be used for learning. Our approach is fundamentally different in that kernel-based reinforcement learning only relies on the sample trajectories of the MDP.…”

Earnings Dispersion, Risk Aversion and Education

“…More details on the return to schooling can be found in Belzil and Hansen (2001). 12 The predicted schooling attainments, along with actual frequencies are found in Table 1, and allow us to evaluate the goodness of¯t. There is clear evidence that our model is capable of¯tting the data well.…”