A policy iteration algorithm for zero-sum stochastic games with mean payoff

Cochet-Terrasson, Jean; Gaubert, Stéphane

doi:10.1016/j.crma.2006.07.011

Cited by 12 publications

(23 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This difficulty was solved first in the deterministic framework in [CTGG99], where it was shown that cycling can be avoided by enforcing a special choice of the bias vector, obtained by a nonlinear projection operation. This approach was then extended to the stochastic framework in [CTG06,ACTDG12]. As a special case of these results, we get that policy iteration is correct and does terminate under much milder conditions than in Theorem 4.1.…”

Section: Theorem 41 (Corollary Of [Hk66]) Algorithm 1 Terminates Anmentioning

confidence: 83%

Generic uniqueness of the bias vector of finite zero-sum stochastic games with perfect information

Akian

Gaubert

Hochart

2018

Journal of Mathematical Analysis and Applications

Self Cite

View full text Add to dashboard Cite

Mean-payoff zero-sum stochastic games can be studied by means of a nonlinear spectral problem. When the state space is finite, the latter consists in finding an eigenpair (u, λ) solution of T (u) = λe + u, where T : R n → R n is the Shapley (or dynamic programming) operator, λ is a scalar, e is the unit vector, and u ∈ R n . The scalar λ yields the mean payoff per time unit, and the vector u, called bias, allows one to determine optimal stationary strategies in the mean-payoff game. The existence of the eigenpair (u, λ) is generally related to ergodicity conditions. A basic issue is to understand for which classes of games the bias vector is unique (up to an additive constant). In this paper, we consider perfect-information zero-sum stochastic games with finite state and action spaces, thinking of the transition payments as variable parameters, transition probabilities being fixed. We show that the bias vector, thought of as a function of the transition payments, is generically unique (up to an additive constant). The proof uses techniques of nonlinear Perron-Frobenius theory. As an application of our results, we obtain an explicit perturbation scheme allowing one to solve degenerate instances of stochastic games by policy iteration.Date: May 24, 2017. 2010 Mathematics Subject Classification. 47J10; 91A20,93E20.

show abstract

Section: Theorem 41 (Corollary Of [Hk66]) Algorithm 1 Terminates Anmentioning

confidence: 83%

Generic uniqueness of the bias vector of finite zero-sum stochastic games with perfect information

Akian

Gaubert

Hochart

2018

Journal of Mathematical Analysis and Applications

Self Cite

View full text Add to dashboard Cite

show abstract

“…Then, using the uniqueness of bias vectors, we deduce from Lemma 3.3 of [17], that if λ k+1 = λ k , then up to an additive constant, v k+1 ≤ v k with equality on the set of critical nodes of T σ k r . This implies that the sequence v k coincides up to an additive constant with the sequence obtained in the algorithm introduced in [19] and developed in [20]. Then, the convergence of Algorithm 21 in a finite number of steps follows from the convergence of the algorithm of [19], [20].…”

Section: Sketch Of Proofmentioning

confidence: 99%

“…This implies that the sequence v k coincides up to an additive constant with the sequence obtained in the algorithm introduced in [19] and developed in [20]. Then, the convergence of Algorithm 21 in a finite number of steps follows from the convergence of the algorithm of [19], [20].…”

Section: Sketch Of Proofmentioning

confidence: 99%

Generic uniqueness of the bias vector of mean payoff zero-sum games

Akian

Gaubert

Hochart

2014

53rd IEEE Conference on Decision and Control

View full text Add to dashboard Cite

Abstract-Zero-sum mean payoff games can be studied by means of a nonlinear spectral problem. When the state space is finite, the latter consists in finding an eigenpair (u, λ) solution of T (u) = λ1 + u where T : R n → R n is the Shapley (dynamic programming) operator, λ is a scalar, 1 is the unit vector, and u ∈ R n . The scalar λ yields the mean payoff per time unit, and the vector u, called the bias, allows one to determine optimal stationary strategies. The existence of the eigenpair (u, λ) is generally related to ergodicity conditions. A basic issue is to understand for which classes of games the bias vector is unique (up to an additive constant). In this paper, we consider perfect information zero-sum stochastic games with finite state and action spaces, thinking of the transition payments as variable parameters, transition probabilities being fixed. We identify structural conditions on the support of the transition probabilities which guarantee that the spectral problem is solvable for all values of the transition payments. Then, we show that the bias vector, thought of as a function of the transition payments, is generically unique (up to an additive constant). The proof uses techniques of max-plus (tropical) algebra and nonlinear Perron-Frobenius theory.

show abstract

“…U g/λ n g/λ n g/λ n kh V λ n +2 /k 2 g = hl/kλ m 1λ n −2 /k 2 g = λ n +2 /k 2 g λ n +2 /k 2 g e/k 2kh X e e e e Y l/k l/k λ 2+n −m h /k 2 g = λ 4 l/1k e/k 2k (11) where the eigenvalue λ of the minplus nonlinear system (10) has been given in Theorem 9…”

Section: Appendixmentioning

confidence: 99%

Piecewise linear concave dynamical systems appearing in the microscopic traffic modeling

Farhi¹,

Goursat²,

Quadrat³

2011

Linear Algebra and its Applications

View full text Add to dashboard Cite

Motivated by microscopic traffic modeling, we analyze dynamical systems which have a piecewise linear concave dynamics not necessarily monotonic. We introduce a deterministic Petri net extension where edges may have negative weights. The dynamics of these Petri nets are uniquely defined and may be described by a generalized matrix with a submatrix in the standard algebra with possibly negative entries, and another submatrix in the minplus algebra. When the dynamics is additively homogeneous, a generalized additive eigenvalue is introduced, and the ergodic theory is used to define a growth rate. In the traffic example of two roads with one junction, we compute explicitly the eigenvalue and we show, by numerical simulations, that these two quantities (the additive eigenvalue and the average growth rate) are not equal, but are close to each other. With this result, we are able to extend the well-studied notion of fundamental traffic diagram (the average flow as a function of the car density on a road) to the case of roads with a junction and give a very simple analytic approximation of this diagram where four phases appear with clear traffic interpretations. Simulations show that the fundamental diagram shape obtained is also valid for systems with many junctions

show abstract

A policy iteration algorithm for zero-sum stochastic games with mean payoff

Cited by 12 publications

References 15 publications

Generic uniqueness of the bias vector of finite zero-sum stochastic games with perfect information

Generic uniqueness of the bias vector of finite zero-sum stochastic games with perfect information

Generic uniqueness of the bias vector of mean payoff zero-sum games

Piecewise linear concave dynamical systems appearing in the microscopic traffic modeling

Contact Info

Product

Resources

About