Dynamic fictitious play, dynamic gradient play, and distributed convergence to Nash equilibria

Abstract

We consider a continuous-time form of repeated matrix games in which player strategies evolve in reaction to opponent actions. Players observe each other's actions, but do not have access to other player utilities. Strategy evolution may be of the best response sort, as in fictitious play, or a gradient update. Such mechanisms are known to not necessarily converge. We introduce a form of "dynamic" fictitious and gradient play strategy update mechanisms. These mechanisms use derivative action in processing opponent actions and, in some cases, can lead to behavior converging to Nash equilibria in previously nonconvergent situations. We analyze convergence in the case of exact and approximate derivative measurements of the dynamic update mechanisms. In the ideal case of exact derivative measurements, we show that convergence to Nash equilibrium can always be achieved. In the case of approximate derivative measurements, we derive a characterization of local convergence that shows how the dynamic update mechanisms can converge if the traditional static counterparts do not. We primarily discuss two player games, but also outline extensions to multiplayer games. We illustrate these methods with convergent simulations of the well known Shapley and Jordan counterexamples.

Submodularity in Dynamics and Control of Networked Systems

“…Learning-based equilibriumseeking algorithms in both discrete [97] and continuous [122] settings provide a possible approach for tightening the error bounds and modeling the interaction over an extended time period. Investigating whether the supermodularity of the objective function for the system can be exploited to improve the bounds on these algorithms is an open problem.…”

State based potential games

“…With a broad set of existing results for learning in potential games (Arslan & Shamma, 2004;Fudenberg & Levine, 1998;Marden, Arslan, & Shamma, 2009bMarden, Young, Arslan, & Shamma, 2009;Shamma & Arslan, 2005;Young, 1998Young, , 2005Young, , 1993, the primary focus of this work is on the development of methodologies for designing the interaction framework as a potential game while meeting constraints and objectives relevant to multiagent systems, e.g., locality of agent objective functions, and efficiency guarantees for resulting equilibria, among many others. Unfortunately, the framework of potential games is not broad enough to meet this diverse set of challenges as several limitations are beginning to emerge.…”

“…The framework of state based potential games is rich enough to overcome the aforementioned limitations as highlighted in Section 7. Interestingly, state based potential games can be thought of in a complimentary fashion to recent work in distributed learning algorithms (Pradelski & Young, 2012;Shamma & Arslan, 2005;Young, 2009) where an underlying state space is introduced into the learning environment to help coordinate behavior. For example, in Young (2009) the authors introduce moods for each agent that impacts the agent's behavior.…”

Probabilistic Interpretations of Integrability for Game Dynamics

“…It would be much less reasonable in alternative models in which agents make simple forecasts about the likely directions of change in the performances of their actions before deciding which action to play. The use of such forecasts can generate dynamics with excellent convergence properties-see Shamma and Arslan (2005) and Arslan and Shamma (2006). 3 For background on population games and evolutionary dynamics, see Sandholm (2010).…”