Morton and Wecker (1977) stated that the value iteration algorithm solves a dynamic program's policy function faster than its value function when the limiting Markov chain is ergodic. I show that their proof is incomplete, and provide a new proof of this classic result. I use this result to accelerate the estimation of Markov decision processes and the solution of Markov perfect equilibria.Keywords. Markov decision process, Markov perfect equilibrium, strong convergence, relative value iteration, dynamic discrete choice, nested fixed point, nested pseudo-likelihood.JEL classification. C01, C13, C15, C61, C63, C65.1. The empirical likelihood of a dynamic model depends only on the dynamic program's policy function, not its value function.2. The policy function depends only on the value function's relative differences, not its absolute level.3. The value function's relative differences converge faster than its level under repeated Bellman contractions when the underlying stochastic process is ergodic. This is called strong convergence. Morton and Wecker (1977) discovered this strong convergence property, but their proof is wanting. At one point, they implicitly replace a close-to-optimal policy with the optimal policy, which is unwarranted. I provide a new proof, based on a straightforward envelope theorem argument.
Markov decision processI consider a dynamic program with discrete time periods, an infinite planning horizon, a finite state space, and an uncountable, compact action space. Taking a specific action in a specific state yields a specific utility. The goal is to determine the actions that yield the Robert L. Bray: r-bray@kellogg.northwestern.edu I would like to thank Victor Aguirregabiria, Achal Bassamboo, Thomas Bray, Seyed Emadi, Soheil Ghili, Arvind Magesan, John Rust, Dennis Zhang, and three anonymous referees for their helpful comments.