Linear-quadratic discrete-time dynamic potential games

Мазалов, Владимир; Rettieva, Anna N.; Avrachenkov, Konstantin

doi:10.1134/s0005117917080136

Cited by 5 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In stochastic optimal control, the MPG model dates back to Dechert and O'Donnell (2006); González-Sánchez and Hernández-Lerma (2013). More recent studies include Zazo et al (2016); Mazalov et al (2017); Macua et al (2018); Mguni et al (2018) and all of these studies focus on systems with known dynamics. MPGs have also attracted attention in multi-agent RL.…”

Section: Related Workmentioning

confidence: 99%

Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence

Ding¹,

Wei²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

We examine global non-asymptotic convergence properties of policy gradient methods for multiagent reinforcement learning (RL) problems in Markov potential games (MPG). To learn a Nash equilibrium of an MPG in which the size of state space and/or the number of players can be very large, we propose new independent policy gradient algorithms that are run by all players in tandem. When there is no uncertainty in the gradient evaluation, we show that our algorithm finds an -Nash equilibrium with O(1/ 2 ) iteration complexity which does not explicitly depend on the state space size. When the exact gradient is not available, we establish O(1/ 5 ) sample complexity bound in a potentially infinitely large state space for a sample-based algorithm that utilizes function approximation. Moreover, we identify a class of independent policy gradient algorithms that enjoys convergence for both zero-sum Markov games and Markov cooperative games with the players that are oblivious to the types of games being played. Finally, we provide computational experiments to corroborate the merits and the effectiveness of our theoretical developments.

show abstract

Section: Related Workmentioning

confidence: 99%

Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence

Ding¹,

Wei²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We will discuss methods for static games, including general Nash equilibrium problems (GNEP) [30], Newton's method [29], Nikaido-Isoda relaxation algorithms [8,15,41,42,44] and extremum seeking [32][33][34]. We will also discuss methods for special dynamic games, such as linear-quadratic games [11,20,24,25,40,45], potential games [35][36][37]49,61,63] and zero-sum games [58,59]. Finally, we will discuss general methods based on Pontryagin's maximum principle [4,7,43].…”

Section: Related Workmentioning

confidence: 99%

“…In a potential game, a single potential function can be used to describe the marginal costs for each player [18,[35][36][37]49,61,63]. Based on this property, potential games can often be solved using methods of single-agent optimization or optimal control.…”

Section: Related Workmentioning

confidence: 99%

Newton’s Method, Bellman Recursion and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Lamperski

2021

Dyn Games Appl

View full text Add to dashboard Cite

Dynamic games arise when multiple agents with differing objectives control a dynamic system. They model a wide variety of applications in economics, defense, energy systems and etc. However, compared to single-agent control problems, the computational methods for dynamic games are relatively limited. As in the single-agent case, only specific dynamic games can be solved exactly, so approximation algorithms are required. In this paper, we show how to extend the Newton step algorithm, the Bellman recursion and the popular differential dynamic programming (DDP) for single-agent optimal control to the case of fullinformation nonzero sum dynamic games. We show that the Newton's step can be solved in a computationally efficient manner and inherits its original quadratic convergence rate to open-loop Nash equilibria, and that the approximated Bellman recursion and DDP methods are very similar and can be used to find local feedback O(ε 2 )-Nash equilibria. Numerical examples are provided. KeywordsNoncooperative dynamic games • Open-loop Nash equilibrium • Feedback Nash equilibrium • Newton's method • Differential dynamic programming B Bolei Di

show abstract

“…Several works focus on the solution to dynamic potential games. Potential games are more tractable than general dynamic games, as they can be solved using methods from single-agent optimal control [16], [17], [18], [19], [20], [21]. However, potential games satisfy restrictive symmetry conditions.…”

Section: A Related Workmentioning

confidence: 99%

Newton’s Method and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Lamperski

2019

2019 IEEE 58th Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

Dynamic games arise when multiple agents with differing objectives choose control a dynamic system. They model a wide variety of applications in economics, defense, energy systems and etc. However, compared to single-agent control problems, the computational methods for dynamic games are relatively limited. As in the single-agent case, only specific dynamic games can be solved exactly, and so approximation algorithms are required. In this paper, we show how to extend a recursive Newton's algorithm and the popular differential dynamic programming (DDP) for single-agent optimal control to the case of full-information non-zero sum dynamic games. In the singleagent case, convergence of DDP is proved by comparison with Newton's method, which converges locally at a quadratic rate. We show that the iterates of Newton's method and DDP are sufficiently close for the DDP to inherit the quadratic convergence rate of Newton's method. We also prove both methods result in an open-loop Nash equilibrium and a local feedback O(ε 2 )-Nash equilibrium. Numerical examples are provided.

show abstract

Linear-quadratic discrete-time dynamic potential games

Cited by 5 publications

References 7 publications

Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence

Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence

Newton’s Method, Bellman Recursion and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Newton’s Method and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Contact Info

Product

Resources

About