Analyzing Multi-agent Reinforcement Learning Using Evolutionary Dynamics

Hoen, Pieter Jan't; Tuyls, Karl

doi:10.1007/978-3-540-30115-8_18

Cited by 9 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section we will show the dynamics of Cross learning and Q-learning and illustrate them on different examples [40,45,48]. First we discuss the main result of Börgers and Sarin [5], i.e.…”

Section: Relating Evolution and Reinforcement Learningmentioning

confidence: 99%

An Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games

Tuyls

Hoen²,

Vanschoenwinkel

2005

Auton Agent Multi-Agent Syst

Self Cite

115

View full text Add to dashboard Cite

In this paper, we investigate Reinforcement learning (RL) in multi-agent systems (MAS) from an evolutionary dynamical perspective. Typical for a MAS is that the environment is not stationary and the Markov property is not valid. This requires agents to be adaptive. RL is a natural approach to model the learning of individual agents. These Learning algorithms are however known to be sensitive to the correct choice of parameter settings for single agent systems. This issue is more prevalent in the MAS case due to the changing interactions amongst the agents. It is largely an open question for a developer of MAS of how to design the individual agents such that, through learning, the agents as a collective arrive at good solutions. We will show that modeling RL in MAS, by taking an evolutionary game theoretic point of view, is a new and potentially successful way to guide learning agents to the most suitable solution for their task at hand. We show how evolutionary dynamics (ED) from Evolutionary Game Theory can help the developer of a MAS in good choices of parameter settings of the used RL algorithms. The ED essentially predict the equilibriums outcomes of the MAS where the agents use individual RL algorithms. More specifically, we show how the ED predict the learning trajectories of Q-Learners for iterated games. Moreover, we apply our results to (an extension of) the COllective INtelligence framework (COIN). COIN is a proved engineering approach for learning of cooperative tasks in MASs. The utilities of the agents are re-engineered to contribute to the global utility. We show how the improved results for MAS RL in COIN, and a developed extension, are predicted by the ED.

show abstract

Section: Relating Evolution and Reinforcement Learningmentioning

confidence: 99%

An Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games

Tuyls

Hoen²,

Vanschoenwinkel

2005

Auton Agent Multi-Agent Syst

Self Cite

115

View full text Add to dashboard Cite

show abstract

“…Many such tools are particularly suited to only certain learning methods, and only a few offer a common framework for multiple learning techniques. Evolutionary game theory is the main tool in this second category, and it was successfully used to study the properties of cooperative coevolution [78,296], to visualize basins of attraction to Nash equilibria for cooperative coevolution [191], and to study trajectories of concurrent Q-learning processes [271,263]. Another tool for modeling and predicting the dynamics of concurrent multi-agent learners was recently proposed in Vidal and Durfee [276,277].…”

Section: The Dynamics Of Learningmentioning

confidence: 99%

Cooperative Multi-Agent Learning: The State of the Art

Panait

Luke

2005

Auton Agent Multi-Agent Syst

982

559

View full text Add to dashboard Cite

Cooperative multi-agent systems are ones in which several agents attempt, through their interaction, to jointly solve tasks or to maximize utility. Due to the interactions among the agents, multi-agent problem complexity can rise rapidly with the number of agents or their behavioral sophistication. The challenge this presents to the task of programming solutions to multi-agent systems problems has spawned increasing interest in machine learning techniques to automate the search and optimization process.We provide a broad survey of the cooperative multi-agent learning literature. Previous surveys of this area have largely focused on issues common to specific subareas (for example, reinforcement learning or robotics). In this survey we attempt to draw from multi-agent learning work in a spectrum of areas, including reinforcement learning, evolutionary computation, game theory, complex systems, agent modeling, and robotics.We find that this broad view leads to a division of the work into two categories, each with its own special issues: applying a single learner to discover joint solutions to multi-agent problems (team learning), or using multiple simultaneous learners, often one per agent (concurrent learning). Additionally, we discuss direct and indirect communication in connection with learning, plus open issues in task decomposition, scalability, and adaptive dynamics. We conclude with a presentation of multi-agent learning problem domains, and a list of multi-agent learning resources.

show abstract