Average Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

Abstract

This paper presents sufficient conditions for the existence of stationary optimal policies for average cost Markov decision processes with Borel state and action sets and weakly continuous transition probabilities. The one-step cost functions may be unbounded, and the action sets may be noncompact. The main contributions of this paper are: (i) general sufficient conditions for the existence of stationary discount optimal and average cost optimal policies and descriptions of properties of value functions and sets of optimal actions, (ii) a sufficient condition for the average cost optimality of a stationary policy in the form of optimality inequalities, and (iii) approximations of average cost optimal actions by discount optimal actions.Key words: Markov decision process; average cost per unit time; optimality inequality; optimal policy MSC2000 subject classification: Primary: 90C40; secondary: 90C39 OR/MS subject classification: Primary: dynamic programming/optimal control; secondary: Markov, infinite state 1. Introduction. This paper provides sufficient conditions for the existence of stationary optimal policies for average cost Markov decision processes (MDPs) with Borel state and action sets and weakly continuous transition probabilities. The cost functions may be unbounded, and the action sets may be noncompact. The main contributions of this paper are: (i) general sufficient conditions for the existence of stationary discount optimal and average cost optimal policies and descriptions of properties of value functions and sets of optimal actions (Theorems 1, 3, and 4), (ii) a new sufficient condition for average cost optimality based on optimality inequalities (Theorem 2), and (iii) approximations of average cost optimal actions by discount optimal actions (Theorem 5).For infinite-horizon MDPs, there are two major criteria: average costs per unit time and expected total discounted costs. The former is typically more difficult to analyze. The so-called vanishing discount factor approach is often used to approximate average costs per unit time by normalized expected total discounted costs. The literature on average cost MDPs is vast. Most of the earlier results are surveyed in Arapostathis et al. [1]. Here, we mention just a few references.For finite-state and action sets, Derman [10] proved the existence of stationary average cost optimal policies. This result follows from Blackwell [6] and it also was independently proved by Viskov and Shiryaev [31]. When either the state set or action set is infinite, even -optimal policies may not exist for some > 0; Ross [25], Dynkin and Yushkevich [11, Chapter 7], Feinberg [12, §5]. For a finite-state set and compact action sets, optimal policies may not exist; Bather [2], Chitashvili [9], and Dynkin and Yushkevich [11, Chapter 7].For MDPs with finite-state and action sets, there exist stationary policies satisfying optimality equations (see Dynkin and Yushkevich [11, Chapter 7], where these equations are called canonical), and, furthermore, any stationary policy satisfy...

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

“…Thus for an MDP an initial state x is considered instead of the initial distribution p. In fact, this MDP possesses a special property that action sets at all the states are equal. For MDPs, Feinberg et al [14] provides general conditions for the existence of optimal policies, validity of optimality equations, and convergence of value iterations. Here we formulate these conditions for an MDP whose action sets in all states are equal.…”

“…According to Feinberg et al [14,Corollary 3.2], the real-valued function ψ(x) = inf a∈A c(x, a), x ∈ X <+∞ , with values in R, is inf-compact on X <+∞ . Furthermore, (6.2) implies that X <+∞ ψ(x)z (n) (dx) ≤ λ, n = 1, 2, .…”

Asymptotic Optimality and Rates of Convergence of Quantized Stationary Policies in Stochastic Control

“…It is well known that the set of deterministic stationary policies contains optimal policies for a large class of infinite horizon discounted cost problems (see, e.g., [6], [14]) and average cost optimal control problems (see, e.g., [1], [14]). …”

Finite state approximations of Markov decision processes with general state and action spaces

“…It is well known that the set of deterministic stationary policies contains an optimal policy for a large class of infinite horizon discounted cost problems (see, e.g., [4], [7]) and average cost optimal control problems (see, e.g., [4]). …”