Robustness inequality for Markov control processes with unbounded costs

Abstract

We study the stability of an optimal control of general Markov chains under perturbations of the transition probabilities. The criterion of optimality is the expected total discounted cost with a random rate of discounting. We give upper bounds for the stability index which are expressed in terms of the weighted total variation distance between the transition probabilities of an original and a perturbed processes. In addition, we show how the inequalities found can be applied to estimate the robustness of the optimal control for some controlled queues and for a certain consumption-investment process. Mathematics Subject Classification: 90C40, 90C31

About stability of risk-seeking optimal stopping

“…The "quantitative estimation of stability of optimal control" is understood in the same sense as, for example, in the works [10,11,12,19], where the stability of some classes of Markov decision processes (MDP's) was investigated, and "stability inequalities" were obtained for discounted and average criteria on an infinite time interval.…”

“…The "stability estimation problem" considered in this paper is interpreted as follows (compare it with [7,10,11,12,19,24]). Let the transition probability p of the Markov process {x t } for which one optimizes a stopping rules:…”

Estimates for perturbations of general discounted Markov control chains

“…The value ∆ * * α (·) was called in [4] and [5] the stability index, but we will name it the index of perturbations because the word "stability" has several meanings. Now a natural task is to obtain upper bounds for ∆ * * α (x).…”

“…The first aim of this work was to obtain bounds for ∆ * * α (x) better than those in [1,4,5], using more elementary estimates, obtained earlier by the authors in [11] for uncontrolled Markov chains. Now let F denote the set of all stationary control policies and suppose that there exists a stationary optimal policy f * , corresponding to Q (see Section 2 for definitions).…”

“…Under certain assumptions proposed in the literature (see, for instance, Assumptions 1 and 2 in [4]), the following basic equalities hold: (3) where V α (x, π) is the total expected discounted cost defined for the approximating DMCP corresponding to Q. Thus, under (3), we have the following identity:…”