Finite state approximations of Markov decision processes with general state and action spaces

Abstract

The purpose of this paper is to prove existence of an ε-equilibrium point in a dynamic Nash game with Borel state space and long-run time average cost criteria for the players. The idea of the proof is first to convert the initial game with ergodic costs to an "equivalent" game endowed with discounted costs for some appropriately chosen value of the discount factor, and then to approximate the discounted Nash game obtained in the first step with a countable state space game for which existence of a Nash equilibrium can be established. From the results of Whitt we know that if for any ε > 0 the approximation scheme is selected in an appropriate way, then Nash equilibrium strategies for the approximating game are also ε-equilibrium strategies for the discounted game constructed in the first step. It is then shown that these strategies constitute an ε-equilibrium point for the initial game with ergodic costs as well. The idea of canonical triples, introduced by Dynkin and Yushkevich in the control setting, is adapted here to the game situation. 1. Introduction. We are considering a two-person Markov game over an infinite time horizon. The state space E of the process {x t } ∞ t=0 controlled by the players is taken to be a Borel space E equipped with the Borel σ-algebra E. The action spaces U 1 and U 2 of player 1 and 2, respectively, are compact subsets of some metric spaces. Let U i denote the Borel σ-algebra on U i , and let P(U i) denote the set of all probability measures on (U i , U i), 1991 Mathematics Subject Classification: Primary 90D10, 90D20; Secondary 90D05, 93E05.

Energy Management in Data Centers with Server Setup Delay: A Semi-MDP Approximation

“…This technique merges the states bearing similar transition probabilities and rewards into one meta-state to obtain a smaller MDP. Several works have been reported on the upper bound derivation for deviation of the total aggregated MDP reward from that of the original MDP in both discounted ( [23], [24]) and non-discounted [25] MDPs. However, to our best knowledge, there exists no reported work on state aggregation for optimal energy management in large-scale data centers that incorporates the stochastic characteristics of the underlying system.…”