Multiple Objective Nonatomic Markov Decision Processes with Total Reward Criteria

Abstract

We consider a Markov decision process with an uncountable state space and multiple rewards. For each policy, its performance is evaluated by a vector of total expected rewards. Under the standard continuity assumptions and the additional assumption that all initial and transition probabilities are nonatomic, we prove that the set of performance vectors for all policies is equal to the set of performance Ž . vectors for nonrandomized Markov policies. This result implies the existence of Ž . optimal nonrandomized Markov policies for nonatomic constrained Markov decision processes with total rewards. We provide two examples of applications of our results to constrained multiple objective problems in inventory control and finance.

Optimality of Mixed Policies for Average Continuous-Time Markov Decision Processes with Constraints

“…It seems that establishing this characterization result could be quite involving, especially for general CTMDP models in Borel spaces. Instead, like in [12,13] and [36] for discrete-time and continuous-time problems with total undiscounted and discounted criteria and [31] focusing on the performance analysis of queueing networks, we pass the average constrained CTMDP problem from the infinite dimensional framework (in the space of measures) to the finite dimensional framework by investigating the space of performance vectors.…”

Nonatomic total rewards Markov decision processes with multiple criteria

“…This result was established in [7] as a corollary of the following fact: the performance set for nonrandomized Markov policies coincides with Research of this coauthor was partially supported by NSF Grant DMI-9908258 with the performance set for all policies in a nonatomic MDP satisfying continuity and compactness conditions. Continuity and compactness conditions were essential for the proofs in Feinberg and Piunovskiy [7].…”

“…Lemma 9 in Feinberg and Piunovskiy [7] implies that there is a poligy y in the new MDP such that fi"(P7) = Rn(P") and RN+"(p7) = Rn (Pn, T ) Consider a sequence Ek \ 0. We define TI > 0 such that for all n = 1 , .…”

“…Feinberg and Piunovskiy [7] proved that if a nonatomic MDP satisfies continuity and compactness conditions then there exists an optimal nonrandomized Markov policy for a multiple criterion problem with constraints. This result was established in [7] as a corollary of the following fact: the performance set for nonrandomized Markov policies coincides with Research of this coauthor was partially supported by NSF Grant DMI-9908258 with the performance set for all policies in a nonatomic MDP satisfying continuity and compactness conditions. Continuity and compactness conditions were essential for the proofs in Feinberg and Piunovskiy [7].…”

“…To do it, we consider a one-step MDP, introduced in Feinberg and Piunovskiy [7, Section 41, with the state space X, set of actions D, sets U($) of available actions at states z E X , where D is the set of all strategic measures in the original MDP and U ( z ) is the set of all strategic measures in the original MDP such that:(a) the initial distribution is concentrated at z, (b) the policy is nonrandomized at step 0. By Lemma 8 in Feinberg and Piunovskiy[7], sets U ( z ) and the set of all strategic measures U, generated by policies nonrandomized at step 0, are measurable. One can prove also that the graph of U is measurable.…”