Solution and Forecast Horizons for Infinite-Horizon Nonhomogeneous Markov Decision Processes

Abstract

We consider the problem of solving a nonhomogeneous infinite horizon Markov Decision Process (MDP) problem in the general case of potentially multiple optimal first period policies. More precisely, we seek an algorithm that, given a finite subset of the problem's potentially infinite data set, delivers an optimal first period policy. Such an algorithm can thus recursively generate, within a rolling horizon procedure, an infinite horizon optimal solution to the original infinite horizon problem. However it can happen that for a given problem no such algorithm exists. In this case, it is impossible to solve the problem with a finite amount of data. We say such problems fail to be wellposed. Under the assumption of increasing marginal returns in actions (with respect to states) and stochastically increasing states into which the system transitions (with respect to actions), we provide an algorithm that is guaranteed to solve the corresponding nonhomogeneous MDP whenever the problem is well-posed. The algorithm proceeds by discovering, in finite time, a forecast horizon for which an optimal solution delivers an optimal first period policy to the infinite horizon problem. In particular, we show by construction, the existence of a forecast horizon (and hence, a solution horizon) for all such well-posed problems. We illustrate the theory and algorithms developed by solving every well-posed instance of the time-varying version of the classic asset selling problem.

A Linear Programming Approach to Nonstationary Infinite-Horizon Markov Decision Processes

“…This theorem only establishes the existence of iterations K n with the stated property -we cannot tell whether we have reached K n since it is not possible in general to finitely establish optimality of early decisions in nonstationary MDPs [13].…”

“…For each (n, s, a), let J n (s, a) be the set of nodes in stage n + 1 that are reachable on choosing action a in node s in stage n. That is, J n (s, a) = {s ∈ S : p n (s |s, a) > 0}. (13) Then, the hyperarc corresponding to action a ∈ A that emanates from the node representing state s ∈ S in stage n ∈ N has |J n (s, a)| "heads". Furthermore, the flow reaching from node s to node s ∈ J n (s, a) equals p n (s |s, a)x n (s, a).…”

“…This may, at first sight, appear to be a weakness of our algorithm. However, it is in fact due to a feature of problem (D) itself, and more generally, of nonstationary infinite-horizon optimization problems -optimality of a given solution cannot be affirmed with finite computations (see [13]). Fortunately, this does not undermine the validity of Theorem 5.3 as its conclusions are trivially true if x k is optimal for some k and we simply repeat this solution for all subsequent k. The next lemma establishes that the Simplex algorithm produces an improving sequence of basic feasible solutions.…”

“…Nonstationary infinite-horizon Markov decision processes (MDPs) [13] (henceforth called nonstationary MDPs) are one of the most general sequential decision models studied in operations research. Nonstationary MDPs extend the more well-studied stationary MDPs [38,42] by relaxing the restrictive assumption that problem data do not change over time.…”

“…Nonstationary MDPs extend the more well-studied stationary MDPs [38,42] by relaxing the restrictive assumption that problem data do not change over time. From a practical viewpoint, nonstationary MDPs incorporate temporal changes in underlying economic and technological conditions into the decision-making process, and have been used to model problems such as asset selling [13] and stochastic inventory control [14]. They can be described as follows.…”