Learning to Coordinate Using Commitment Sequences in Cooperative Multi-agent Systems

Kapetanakis, Spiros; Kudenko, Daniel; Strens, Malcolm J. A.

doi:10.1007/978-3-540-32274-0_7

Cited by 10 publications

(4 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, in games with stochastic rewards the heuristic may perform poorly. Therefore the authors propose the use of commitment sequences in [10] to allow for stochastic rewards. In a way these commitment sequences are very similar to the exploration phases of ESRL learning, however the former approach strongly depends on synchronous action-selection, whereas ESRL is suited for asynchronous real-life applications as shown in Sect.…”

Section: Related Workmentioning

confidence: 99%

Exploring selfish reinforcement learning in repeated games with stochastic rewards

Verbeeck

Nowé

Parent

et al. 2006

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

In this paper we introduce a new multi-agent reinforcement learning algorithm, called exploring selfish reinforcement learning (ESRL). ESRL allows agents to reach optimal solutions in repeated non-zero sum games with stochastic rewards, by using coordinated exploration. First, two ESRL algorithms for respectively common interest and conflicting interest games are presented. Both ESRL algorithms are based on the same idea, i.e. an agent explores by temporarily excluding some of the local actions from its private action space, to give the team of agents the opportunity to look for better solutions in a reduced joint action space. In a latter stage these two algorithms are transformed into one generic algorithm which does not assume that the type of the game is known in advance. ESRL is able to find the Pareto optimal solution in common interest games without communication. In conflicting interest games ESRL only needs limited communication to learn a fair periodical policy, resulting in a good overall policy. Important to know is that ESRL agents are independent in the sense that they only use their own action choices and rewards to base their decisions on, that ESRL agents are flexible in learning different solution concepts and they can handle both stochastic, possible delayed rewards and asynchronous action selection. A real-life experiment, i.e. adaptive load-balancing of parallel applications is added.K. Verbeeck (B) Computational Modeling Lab (COMO), Vrije Universiteit Brussel, Brussels, Belgium

show abstract

Section: Related Workmentioning

confidence: 99%

Exploring selfish reinforcement learning in repeated games with stochastic rewards

Verbeeck

Nowé

Parent

et al. 2006

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

show abstract

“…First, we apply the algorithms to the cooperative matrix games presented in Figures 1 and 2 (pages 4, 6). These games have received considerable attention from the community (Claus & Boutilier, 1998;Lauer & Riedmiller, 2004;Carpenter & Kudenko, 2005;Kapetanakis et al, 2005;McGlohon & Sen, 2005;Verbeeck et al, 2007) as they remain challenging for state-of-the-art algorithms despite their apparent simplicity. Indeed, even a simple game with two players and few actions can be challenging if agents are unaware of the game and independent (Abdallah & Lesser, 2008).…”

Section: Penalty and Climbing Gamesmentioning

confidence: 99%

Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems

Matignon

Laurent

Fort-Piat

2012

The Knowledge Engineering Review

327

260

View full text Add to dashboard Cite

AbstractIn the framework of fully cooperative multi-agent systems, independent (non-communicative) agents that learn by reinforcement must overcome several difficulties to manage to coordinate. This paper identifies several challenges responsible for the non-coordination of independent agents: Pareto-selection, non-stationarity, stochasticity, alter-exploration and shadowed equilibria. A selection of multi-agent domains is classified according to those challenges: matrix games, Boutilier's coordination game, predators pursuit domains and a special multi-state game. Moreover, the performance of a range of algorithms for independent reinforcement learners is evaluated empirically. Those algorithms are Q-learning variants: decentralized Q-learning, distributed Q-learning, hysteretic Q-learning, recursive frequency maximum Q-value and win-or-learn fast policy hill climbing. An overview of the learning algorithms’ strengths and weaknesses against each challenge concludes the paper and can serve as a basis for choosing the appropriate algorithm for a new domain. Furthermore, the distilled challenges may assist in the design of new learning algorithms that overcome these problems and achieve higher performance in multi-agent applications.

show abstract

“…Only convergence to (possibly suboptimal) Nash equilibria can be guaranteed (Claus and Boutilier, 1998). Games such as the climbing game (Kapetanakis et al, 2003), shown in Figure 2 are generally accepted as hard coordination problems for independent learners 4 . In this game, independent learners may get stuck in the suboptimal Nash equilibrium (a 12 ,a 22 ) with payoff 7.…”

Section: Exploring Selfish Reinforcement Learnersmentioning

confidence: 99%

A reinforcement learning approach to coordinate exploration with limited communication in continuous action games

Rodriguez

Vrancx

Grau

et al. 2016

The Knowledge Engineering Review

View full text Add to dashboard Cite

Learning automata are reinforcement learners belonging to the class of policy iterators. They have already been shown to exhibit nice convergence properties in a wide range of discrete action game settings. Recently, a new formulation for a continuous action reinforcement learning automata (CARLA) was proposed. In this paper, we study the behavior of these CARLA in continuous action games and propose a novel method for coordinated exploration of the joint-action space. Our method allows a team of independent learners, using CARLA, to find the optimal joint action in common interest settings. We first show that independent agents using CARLA will converge to a local optimum of the continuous action game. We then introduce a method for coordinated exploration which allows the team of agents to find the global optimum of the game. We validate our approach in a number of experiments.

show abstract

Learning to Coordinate Using Commitment Sequences in Cooperative Multi-agent Systems

Cited by 10 publications

References 1 publication

Exploring selfish reinforcement learning in repeated games with stochastic rewards

Exploring selfish reinforcement learning in repeated games with stochastic rewards

Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems

A reinforcement learning approach to coordinate exploration with limited communication in continuous action games

Contact Info

Product

Resources

About