Existence and Regularity of a Nonhomogeneous Transition Matrix under Measurability Conditions

Abstract

This paper is about the existence and regularity of the transition probability matrix of a nonhomogeneous continuous-time Markov process with a countable state space. A standard approach to prove the existence of such a transition matrix is to begin with a continuous (in t) and conservative matrix Q(t) = [q ij (t)] of nonhomogeneous transition rates q ij (t), and use it to construct the transition probability matrix. Here we obtain the same result except that the q ij (t) are only required to satisfy a mild measurability condition, and Q(t) may not be conservative. Moreover, the resulting transition matrix is shown to be the minimum transition matrix and, in addition, a necessary and sufficient condition for it to be regular is obtained. These results are crucial in some applications of nonhomogeneous continuous-time Markov processes, such as stochastic optimal control problems and stochastic games, which motivated this work in the first place.

Denumerable continuous-time Markov decision processes with multiconstraints on average costs

“…0. Hence, for each 2 Å m , Theorem 3 in Ye et al (2008) indeed guarantees the existence of a Q process.…”

“…As the existence of a transition function without the continuity condition has been proven in Ye, Guo, and Herna´ndez-Lerma (2008), the multiconstrained problem can be defined well when the costs are bounded below. Under suitable conditions we also show the existence of constrained optimal policies by using a martingale technique.…”

Uniform ergodicity of continuous-time controlled Markov chains: A survey and new results

“…The reader interested in the nonhomogeneous case can consult Appendices B and C in Guo and Hernández-Lerma (2009), as well as the paper (Ye et al 2008). Hence, in what follows, we will consider a homogeneous continuoustime Markov chain x.…”

Total reward criteria for unconstrained/constrained continuous-time Markov decision processes

“…Under Assumption A, Theorem 3 in [25] together with Lemma 3.2 in [8] shows that each Q(π t )-transition function is regular. We denote by p π (s, i, t, j) the associated, unique and honest Q(π t )-transition function, and write the corresponding Markov chain as {x(t)}.…”