Discounted Dynamic Programming

Total Expected Discounted Reward MDPS : Existence of Optimal Policies

“…The values V N (x) is the supremum of the expected total reward over the finite horizon N with the 0 terminal value V 0 . As was shown by Blackwell [3], the functions V N , N = 1, 2 . .…”

“…For the classical dynamic programming problems introduced by Blackwell [3], the reward functions r(x, a) are assumed to be bounded, i.e., |r(x, a)| ≤ K, x ∈ X and a ∈ A(x), for some finite constant K. However, in many operations research applications the reward functions are bounded above, i.e., r(x, a) ≤ K when x ∈ X and a ∈ A(x). For example, in mathematical models of inventory and queueing systems, the one-step holding costs can tend to ∞ as the inventory levels or number of waiting customers increases to ∞.…”

“…In a general situation, the value function V (x) may not be Borel measurable, but it is universally measurable; see [2,3,7] for detail. Since for any fixed policy π the function v π (x) is Borel, -optimal policies may not exist; Blackwell [3,Example 2].…”

“…In a general situation, the value function V (x) may not be Borel measurable, but it is universally measurable; see [2,3,7] for detail. Since for any fixed policy π the function v π (x) is Borel, -optimal policies may not exist; Blackwell [3,Example 2]. However, such policies exist if either the definition of -optimal is changed to the definition of so-called (p, )-optimal policies or the set of policies is expanded by allowing the policies to be universally measurable.…”

A dynamic model integrating demand and supply relationships for agricultural water, applied to determining optimal intertemporal allocation of water in a regional water project

“…Dynamic programming theory for such Markovian cases [Blackwell 1965, Ross 1970, verifies that a search for a policy that is optimal among all policies may be restricted to the search for policy that is optimal among all stationary policies, i.e.. policies that dictate actions only in accordance with the current state independently of the current stage (year) or past history. Hence, a search for an optimal stationary policy engaged in this thesis is in fact a search for a policy that is optimal without qualifications.…”