A selection-mutation model for q-learning in multi-agent systems

Tuyls, Karl; Verbeeck, Katja; Lenaerts, Tom

doi:10.1145/860685.860687

Cited by 36 publications

(86 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This represents the exploration of action by individual i (it is an analogue of mutation in evolutionary biology), and brings the dynamics back into the interior of the state space if it gets too close to the boundary. The second term in the brackets takes the same form as the replicator equation (Hofbauer and Sigmund, 1998;Tuyls et al, 2003); that is, if the expected reinforcement,R i (a), to action a is higher than the average expected reinforcement,…”

Section: Differential Equations For Action Play Probabilitiesmentioning

confidence: 99%

On learning dynamics underlying the evolution of learning rules

Dridi

Lehmann

2014

Theoretical Population Biology

View full text Add to dashboard Cite

In order to understand the development of non-genetically encoded actions during an animal's lifespan, it is necessary to analyze the dynamics and evolution of learning rules producing behavior. Owing to the intrinsic stochastic and frequency-dependent nature of learning dynamics, these rules are often studied in evolutionary biology via agent-based computer simulations. In this paper, we show that stochastic approximation theory can help to qualitatively understand learning dynamics and formulate analytical models for the evolution of learning rules. We consider a population of individuals repeatedly interacting during their lifespan, and where the stage game faced by the individuals fluctuates according to an environmental stochastic process. Individuals adjust their behavioral actions according to learning rules belonging to the class of experience-weighted attraction learning mechanisms, which includes standard reinforcement and Bayesian learning as special cases. We use stochastic approximation theory in order to derive differential equations governing action play probabilities, which turn out to have qualitative features of mutator-selection equations. We then perform agent-based simulations to find the conditions where the deterministic approximation is closest to the original stochastic learning process for standard 2-action 2-player fluctuating games, where interaction between learning rules and preference reversal may occur. Finally, we analyze a simplified model for the evolution of learning in a producer-scrounger game, which shows that the exploration rate can interact in a non-intuitive way with other features of co-evolving learning rules. Overall, our analyses illustrate the usefulness of applying stochastic approximation theory in the study of animal learning.

show abstract

Section: Differential Equations For Action Play Probabilitiesmentioning

confidence: 99%

On learning dynamics underlying the evolution of learning rules

Dridi

Lehmann

2014

Theoretical Population Biology

View full text Add to dashboard Cite

show abstract

“…This system (without the exploration term) is known as bi-matrix replicator equation [20,21]. Its relation to multi-agent learning has been examined in [6,8,[22][23][24]. Before proceeding further, we elaborate on the connection between the rest-points of the replicator system Eqs.…”

Section: B Two-agent Learningmentioning

confidence: 99%

Dynamics of BoltzmannQlearning in two-player two-action games

Kianercy

Galstyan

2012

Phys. Rev. E

View full text Add to dashboard Cite

We consider the dynamics of Q learning in two-player two-action games with a Boltzmann exploration mechanism. For any nonzero exploration rate the dynamics is dissipative, which guarantees that agent strategies converge to rest points that are generally different from the game's Nash equlibria (NEs). We provide a comprehensive characterization of the rest point structure for different games and examine the sensitivity of this structure with respect to the noise due to exploration. Our results indicate that for a class of games with multiple NEs the asymptotic behavior of learning dynamics can undergo drastic changes at critical exploration rates. Furthermore, we demonstrate that, for certain games with a single NE, it is possible to have additional rest points (not corresponding to any NE) that persist for a finite range of the exploration rates and disappear when the exploration rates of both players tend to zero.

show abstract

“…Namely, if one associates a particular biological trait with each pure strategy, then the adaptive learning of (possibly mixed) strategies in multi-agent settings is analogous to competitive dynamics of mixed population, where the species evolve according to their relative fitness in the population. This framework has been used successfully to study various interesting features of adaptive dynamics of learning agents [7,16,19,20,[24][25][26].…”

Section: Introductionmentioning

confidence: 99%

Continuous strategy replicator dynamics for multi-agent Q-learning

Galstyan

2011

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

The problem of multi-agent learning and adaptation has attracted a great deal of attention in recent years. It has been suggested that the dynamics of multi agent learning can be studied using replicator equations from population biology. Most existing studies so far have been limited to discrete strategy spaces with a small number of available actions. In many cases, however, the choices available to agents are better characterized by continuous spectra. This paper suggests a generalization of the replicator framework that allows to study the adaptive dynamics of Q-learning agents with continuous strategy spaces. Instead of probability vectors, agents' strategies are now characterized by probability measures over continuous variables. As a result, the ordinary differential equations for the discrete case are replaced by a system of coupled integral-differential replicator equations that describe the mutual evolution of individual agent strategies. We derive a set of functional equations describing the steady state of the replicator dynamics, examine their solutions for several two-player games, and confirm our analytical results using simulations.

show abstract

A selection-mutation model for q-learning in multi-agent systems

Cited by 36 publications

References 2 publications

On learning dynamics underlying the evolution of learning rules

On learning dynamics underlying the evolution of learning rules

Dynamics of BoltzmannQlearning in two-player two-action games

Continuous strategy replicator dynamics for multi-agent Q-learning

Contact Info

Product

Resources

About