Zero-Sum Markov Games with Impulse Controls

Abstract

In this paper we consider a zero-sum Markov stopping game on a general state space with impulse strategies and infinite time horizon discounted payoff where the state dynamics is a weak Feller-Markov process. One of the key contributions is our analysis of this problem based on "shifted" strategies, thereby proving that the original game can be practically restricted to a sequence of Dynkin's stopping games without affecting the optimalty of the saddle-point equilibria and hence completely solving some open problems in the existing literature. Under two quite general (weak) assumptions, we show the existence of the value of the game and the form of saddle-point (optimal) equilibria in the set of shifted strategies. Moreover, our methodology is different from the previous techniques used in the existing literature and is based on purely probabilistic arguments. In the process, we establish an interesting property of the underlying Feller-Markov process and the impossibility of infinite number of impulses in finite time under saddle-point strategies which is crucial for the verification result of the corresponding Isaacs-Bellman equations.

On an approximation of average cost per unit time impulse control of Markov processes

“…We restrict here practically to impulses which at time τ i+1 depend only on the behavior of the controlled process after time τ i . Using the procedure as in section 7 of [2] we can show that extension to general impulse strategies (depending on the whole history) does not change the value of w. Then similarly as in the proof of Lemma 2.1 for each n we have (2.24)…”

“…where V n denotes impulse strategy consisting of at most n impulses. In fact, we have first (3.18) for so called shifted impulse strategies (see the definition in section 2 of [2]), which then can be extended using technics of section 7 of [2] to any impulse strategies consisting of n impulses. Notice that we can restrict ourselves to stopping times such that (3.12) is satisfied.…”

“…where ρ is the metric on E. Furthermore for a given compact set B(K, R) = {y : ρ(y, K) ≤ R} for a given ε > 0 and γ > 0 by Proposition 6.4 of [2] there is κ > 0 such that By continuity of w it is uniformly continuous in a compact set so that for a sufficiently small γ we have that By Corollary 1 of [28] we have that z T ∈ C(E) and therefore an optimal stopping τ T time is of the form (4.31) τ T = inf {s ≥ 0 : z T −s (X s ) ≥ w(X s )} .…”

Neuro-Optimal Event-Triggered Impulsive Control for Stochastic Systems via ADP

“…Consequently, the regular time-based transition matrices (P and P ) can be used to derive the evolution of the probability distribution at the prefixed time sequence, but not the evolution of the probability distribution across the impulsive actions. In summary, these facts indicate that the restrictions of the regular time-based transition matrices and the variable impulsive control cycles of the impulsive controller show no compatibility and conflict with each other, which causes the traditional impulsive control methods [24], [25], [26], [27] complicated and highly specialized with low generality and uniformity. Noticing that the "arrival of the impulsive action" can be treated as an event, therefore to address the above issues, a novel generalevent-based impulsive transition matrix is needed to represent the probability distribution evolving characteristics across the impulsive actions.…”

“…Miller et al [25] develop the martingale representation of the stochastic systems subject to joint impulsive and gradual controls, while constructing the optimal strategy based on the DP equation. Basu and Stettner [26] provide the optimal impulsive controller design schemes for zero-sum games under several weak assumptions and weak Feller conditions. Dufour and Piunovskiy [27] study continuous-time stochastic systems on a general Borel state space governed by both impulsive and regular controllers to This work is licensed under a Creative Commons Attribution 4.0 License.…”