Deterministic optimal policies for Markov control processes with pathwise constraints

Abstract

This paper deals with discrete-time Markov control processes in Borel spaces with unbounded rewards. Under suitable hypotheses, we show that a randomized stationary policy is optimal for a certain expected constrained problem (ECP) if and only if it is optimal for the corresponding pathwise constrained problem (pathwise CP). Moreover, we show that a certain parametric family of unconstrained optimality equations yields convergence properties that lead to an approximation scheme which allows us to obtain constrained optimal policies as the limit of unconstrained deterministic optimal policies. In addition, we give sufficient conditions for the existence of deterministic policies that solve these constrained problems.

Variance-minimization of Markov control processes with pathwise constraints

“…Then V() ¼ min à 0 (Ã). (ii) (see [10,Theorem 4.8(b), (c) and Remark 2.2(ii) above. For each à 0, let f à 2 F be a stationary deterministic policy such that for every x 2 X, the action f à (x) 2 A(x) attains the maximum in the right-hand side of (16).…”

“…The following example is the sequel of the one given in [10,Section 6]). Suppose that the reward-per-stage function (51) and the cost-per-stage function (52) satisfy r 1 ¼ 1, r 2 ¼ 2, e ¼ 10 and c 1 ¼ c 2 ¼ 1, respectively.…”

“…In [10], we give conditions for the existence of deterministic (as opposed to randomized) optimal policies for the pathwise CP. These optimal policies, however, may have an arbitrarily bad behaviour for large, but finite, lengths of time.…”

“…The works [8][9][10][13][14][15] have shown the existence of optimal policies for several classes of MCPs with pathwise constraints, that is, policies that maximize a certain long-run pathwise average reward subjet to a constraint on a long-run pathwise average cost. In [10], we give conditions for the existence of deterministic (as opposed to randomized) optimal policies for the pathwise CP.…”