Discounted continuous-time constrained Markov decision processes in Polish spaces

Abstract

This paper is devoted to studying constrained continuous-time Markov decision processes (MDPs) in the class of randomized policies depending on state histories. The transition rates may be unbounded, the reward and costs are admitted to be unbounded from above and from below, and the state and action spaces are Polish spaces. The optimality criterion to be maximized is the expected discounted rewards, and the constraints can be imposed on the expected discounted costs. First, we give conditions for the nonexplosion of underlying processes and the finiteness of the expected discounted rewards/costs. Second, using a technique of occupation measures, we prove that the constrained optimality of continuous-time MDPs can be transformed to an equivalent (optimality) problem over a class of probability measures. Based on the equivalent problem and a so-calledw-weak convergence of probability measures developed in this paper, we show the existence of a constrained optimal policy. Third, by providing a linear programming formulation of the equivalent problem, we show the solvability of constrained optimal policies. Finally, we use two computable examples to illustrate our main results. . This reprint differs from the original in pagination and typographic detail. 1 2 X. GUO AND X. SONG [13,15,34,36, 42], and (iv) constrained continuous-time MDPs with a Polish state space [11]. A review of these references shows that most of the related literature is concentrated with the first three groups. To the best of our knowledge, the fourth group is addressed only in [11] for the average criteria. Concerning group (i), the existence and algorithms of constrained optimal policies are given in [6][7][8][9][10] for variant discounted criteria when states and actions are finite, in [1,25,37] for the discounted criteria and denumerable states, and in [1,2,23,37,38] for the average criteria and denumerable states. Also, the existence of constrained optimal policies and linear programming formulation for group (ii) are given in [19,33] for the discounted criteria and in [20,29,33] for the average criteria. Although group (iii) has been studied in [13,15,34,36, 42], the references [13,15,34,36, 42] deal with the case of a single constraint, the transition rates in [34] are assumed to be bounded, and the assumption of denumerable states in these references cannot be dropped. On the other hand, as mentioned above, constrained MDPs in Polish spaces are also studied in [19,20,29,33] for the discretetime case and in [11] for the continuous-time case. However, the reward and cost functions in [29] are assumed to be all bounded, and all cost functions in [11,19,20,33] are assumed to be essentially nonnegative. Further, such nonnegativeness assumption cannot be removed because it is required for the use of the standard weak convergence of probability measures. This in turn implies that the constrained optimality problem of minimizing nonnegative costs in [11,19,20] with constraints imposed on other nonnegative costs cannot be transformed to an equ...

Absorbing Continuous-Time Markov Decision Processes with Total Cost Criteria

“…The primitives of a CTMDP are the elements [15], [24], [25], [30] {S, A, (A(x) ⊆ A, x ∈ S), q(· | x, a), γ }, where • S (state space) is a nonempty Polish space endowed with the Borel σ -algebra B(S);…”

“…Below we briefly recall it in order to define the necessary terminologies; see [15], [24], [25], [30] for more details.…”

“…. , as considered in [31, Definition 3.1] (see also [15,Definition 3.4]). So, by Theorem 3.2 of [31] (see also Theorem 3.5(a) of [15]), we see, for each S ∈ B(S),…”

“…. , v ex,k−1 ) ∈ V is an extreme point of V (recalling also (15)). Therefore, by the inductive supposition applied to the auxiliary CTMDP model with k − 1 constraints, the extreme point v ex is generated by a deterministic stationary policy, which, by (15), also generates v ex .…”

“…; see the examples in the monographs [13] and [25]. Two standard performance measures of a CTMDP are the (expected) long-run average costs [12], [14], [17], [24], [34], [39] and the (expected) total discounted costs [15], [27], [30]- [32]. The long-run average criteria are not appropriate for CTMDPs with transient behavior because in that case the long-run average costs will be zero for each policy.…”