From Darwin to Poincaré and von Neumann: Recurrence and Cycles in Evolutionary and Algorithmic Game Theory

Boone, Victor; Piliouras, Georgios

doi:10.1007/978-3-030-35389-6_7

Cited by 17 publications

(26 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of works in the past several years show that online no-regret learning dynamics are Poincaré recurrent in repeated static zero-sum games (see, e.g., [24,20,6]). The proof methods for deriving such results crucially rely on the static nature of the game for the reason that the learning dynamics amount to an autonomous dynamical system.…”

Section: Poincaré Recurrence In Autonomous Dynamical Systemsmentioning

confidence: 99%

“…Despite the classical nature of the study of online no-regret learning dynamics in zero-sum games, the actual transient behavior of such dynamics was historically not as understood. However, in the past several years this topic has gained attention with a number of works studying such dynamics in zero-sum games (and variants thereof) with a particular focus on continuous-time analysis [24,25,20,6,32,23,22]. The unifying emergent picture is that the dynamics are "approximately cyclic" in a formal sense known as Poincaré recurrence.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Online Learning in Periodic Zero-Sum Games

Fiez¹,

Sim²,

Skoulakis³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

A seminal result in game theory is von Neumann's minmax theorem, which states that zero-sum games admit an essentially unique equilibrium solution. Classical learning results build on this theorem to show that online no-regret dynamics converge to an equilibrium in a time-average sense in zero-sum games. In the past several years, a key research direction has focused on characterizing the day-to-day behavior of such dynamics. General results in this direction show that broad classes of online learning dynamics are cyclic, and formally Poincaré recurrent, in zero-sum games. We analyze the robustness of these online learning behaviors in the case of periodic zero-sum games with a time-invariant equilibrium. This model generalizes the usual repeated game formulation while also being a realistic and natural model of a repeated competition between players that depends on exogenous environmental variations such as time-of-day effects, week-to-week trends, and seasonality. Interestingly, time-average convergence may fail even in the simplest such settings, in spite of the equilibrium being fixed. In contrast, using novel analysis methods, we show that Poincaré recurrence provably generalizes despite the complex, non-autonomous nature of these dynamical systems.How do online learning dynamics behave if the zero-sum game evolves over time?Clearly, the answer to this question depends critically on how the game is allowed to evolve.

show abstract

Section: Poincaré Recurrence In Autonomous Dynamical Systemsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Online Learning in Periodic Zero-Sum Games

Fiez¹,

Sim²,

Skoulakis³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…RD is the continuous-time variant of the well-known multiplicative weights update (MWU) meta-algorithm [Arora et al, 2012b, Kleinberg et al, 2009, and the seminal dynamics in the areas of mathematical evolution, biology, ecology, and evolutionary game theory [Hofbauer andSigmund, 1998, Weibull, 1997]. In recent years, RD has enjoyed a particularly strong surge in applications to learning in multiplayer games [Boone and Piliouras, 2019, Flokas et al, 2020, Hennes et al, 2020, Nagarajan et al, 2020, Sanders et al, 2018, Skoulakis et al, 2021, Sorin, 2020. Despite its algorithmic simplicity, RD is well-known to minimize external regret (a concept later detailed in Section 3.1), thus yielding time-average convergence to a coarse correlated equilibrium [Mertikopoulos et al, 2018, Sorin, 2009.…”

Section: Replicator Dynamicsmentioning

confidence: 99%

Evolutionary Dynamics and $Φ$-Regret Minimization in Games

Piliouras¹,

Rowland²,

Omidshafiei³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Regret has been established as a foundational concept in online learning, and likewise has important applications in the analysis of learning dynamics in games. Regret quantifies the difference between a learner's performance against a baseline in hindsight. It is wellknown that regret-minimizing algorithms converge to certain classes of equilibria in games; however, traditional forms of regret used in game theory predominantly consider baselines that permit deviations to deterministic actions or strategies. In this paper, we revisit our understanding of regret from the perspective of deviations over partitions of the full mixed strategy space (i.e., probability distributions over pure strategies), under the lens of the previously-established Φ-regret framework, which provides a continuum of stronger regret measures. Importantly, Φ-regret enables learning agents to consider deviations from and to mixed strategies, generalizing several existing notions of regret such as external, internal, and swap regret, and thus broadening the insights gained from regret-based analysis of learning algorithms. We prove here that the well-studied evolutionary learning algorithm of replicator dynamics (RD) seamlessly minimizes the strongest possible form of Φ-regret in generic 2 × 2 games, without any modification of the underlying algorithm itself. We subsequently conduct experiments validating our theoretical results in a suite of 144 2 × 2 games wherein RD exhibits a diverse set of behaviors. We conclude by providing empirical evidence of Φ-regret minimization by RD in some larger games, hinting at further opportunity for Φ-regret based study of such algorithms from both a theoretical and empirical perspective.

show abstract

“…In zero-sum games the dynamics of standard learning algorithms such as gradient descent do not converge to Nash equilibria. Instead, the resultant dynamics may lead to cycling (Mertikopoulos et al, 2018;Vlatakis-Gkaragkounis et al, 2019;Boone and Piliouras, 2019;Balduzzi et al, 2018), divergence (Bailey and Piliouras, 2018;Cheung, 2018), or formally chaotic behaviours (Cheung andPiliouras, 2019, 2020). In the face of such strong negative results for out-of-the-box optimization methods the development of tailored algorithmic solutions is incentivized, e.g.…”

Section: Related Workmentioning

confidence: 99%

Learning in Matrix Games can be Arbitrarily Complex

Andrade¹,

Frongillo²,

Piliouras³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

A growing number of machine learning architectures, such as Generative Adversarial Networks, rely on the design of games which implement a desired functionality via a Nash equilibrium. In practice these games have an implicit complexity (e.g. from underlying datasets and the deep networks used) that makes directly computing a Nash equilibrium impractical or impossible. For this reason, numerous learning algorithms have been developed with the goal of iteratively converging to a Nash equilibrium. Unfortunately, the dynamics generated by the learning process can be very intricate and instances of training failure hard to interpret. In this paper we show that, in a strong sense, this dynamic complexity is inherent to games. Specifically, we prove that replicator dynamics, the continuous-time analogue of Multiplicative Weights Update, even when applied in a very restricted class of games-known as finite matrix games-is rich enough to be able to approximate arbitrary dynamical systems. Our results are positive in the sense that they show the nearly boundless dynamic modelling capabilities of current machine learning practices, but also negative in implying that these capabilities may come at the cost of interpretability. As a concrete example, we show how replicator dynamics can effectively reproduce the well-known strange attractor of Lonrenz dynamics (the "butterfly effect", Fig 1) while achieving no regret.

show abstract

From Darwin to Poincaré and von Neumann: Recurrence and Cycles in Evolutionary and Algorithmic Game Theory

Cited by 17 publications

References 33 publications

Online Learning in Periodic Zero-Sum Games

Online Learning in Periodic Zero-Sum Games

Evolutionary Dynamics and $Φ$-Regret Minimization in Games

Learning in Matrix Games can be Arbitrarily Complex

Contact Info

Product

Resources

About