PettingZoo: Gym for Multi-Agent Reinforcement Learning

Terry, Justin K.; Black, Benjamin; Grammel, Nathaniel; Jayakumar, Mario; Hari, Ananth; Sullivan, Ryan; Santos, Luís; Pérez, Rodrigo; Horsch, Caroline; Dieffendahl, Clemens; Williams, Niall; Lokesh, Yashas; Ravi, Praveen

doi:10.48550/arxiv.2009.14471

Cited by 33 publications

(44 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we evaluate the proposed CAC using two environments: 1) the cooperative navigation task in [8], which is built on the OpenAI Gym platform [62], and 2) the pursuit-evasion game [31], which is built on the PettingZoo platform [46].…”

Section: Numerical Resultsmentioning

confidence: 99%

“…In this case, the shared component θ s i can serve as such a common feature extractor. From many empirical results on RL [45,31,46], a robust feature extractor is highly beneficial for agents to improve their stability and practical performance. Moreover, in a closely related area of federated learning, it has also been observed that partially personalized models improve the generalization power among different agents, leading to better feature representations [47,48,49].…”

Section: The Proposed Formulationmentioning

confidence: 99%

See 1 more Smart Citation

Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees

Zeng¹,

Chen²,

García³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multi-agent reinforcement learning (MARL) has attracted much research attention recently. However, unlike its single-agent counterpart, many theoretical and algorithmic aspects of MARL have not been well-understood. In this paper, we study the emergence of coordinated behavior by autonomous agents using an actor-critic (AC) algorithm. Specifically, we propose and analyze a class of coordinated actor-critic algorithms (CAC) in which individually parametrized policies have a shared part (which is jointly optimized among all agents) and a personalized part (which is only locally optimized). Such kind of partially personalized policy allows agents to learn to coordinate by leveraging peers' past experience and adapt to individual tasks. The flexibility in our design allows the proposed MARL-CAC algorithm to be used in a fully decentralized setting, where the agents can only communicate with their neighbors, as well as a federated setting, where the agents occasionally communicate with a server while optimizing their (partially personalized) local models. Theoretically, we show that under some standard regularity assumptions, the proposed MARL-CAC algorithm requires O( − 5 2 ) samples to achieve an -stationary solution (defined as the solution whose squared norm of the gradient of the objective function is less than ). To the best of our knowledge, this work provides the first finite-sample guarantee for decentralized AC algorithm with partially personalized policies.

show abstract

Section: Numerical Resultsmentioning

confidence: 99%

Section: The Proposed Formulationmentioning

confidence: 99%

Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees

Zeng¹,

Chen²,

García³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…While multi-agent reinforcement learning (MARL) is a well-established branch of Deep RL, most learning algorithms and environments proposed have targeted a relatively small number of agents 17,41 . It is common to see environments with less dozens of agents 1,30,52,67 , with 2-agent and 4-agent environments being particular popular for the study of competitive, self-play settings 2,23,36 . Collective intelligence observed in nature, however, rely on a much larger number of individuals than typically studied in MARL, involving population sizes from thousands to million.…”

Section: Multi-agent Learningmentioning

confidence: 99%

Collective Intelligence for Deep Learning: A Survey of Recent Developments

Ha¹,

Tang²

2021

Preprint

View full text Add to dashboard Cite

In the past decade, we have witnessed the rise of deep learning to dominate the field of artificial intelligence. Advances in artificial neural networks alongside corresponding advances in hardware accelerators with large memory capacity, together with the availability of large datasets enabled researchers and practitioners alike to train and deploy sophisticated neural network models that achieve state-of-the-art performance on tasks across several fields spanning computer vision, natural language processing, and reinforcement learning. However, as these neural networks become bigger, more complex, and more widely used, fundamental problems with current deep learning models become more apparent. State-of-the-art deep learning models are known to suffer from issues that range from poor robustness, inability to adapt to novel task settings, to requiring rigid and inflexible configuration assumptions. Ideas from collective intelligence, in particular concepts from complex systems such as self-organization, emergent behavior, swarm optimization, and cellular systems tend to produce solutions that are robust, adaptable, and have less rigid assumptions about the environment configuration. It is therefore natural to see these ideas incorporated into newer deep learning methods. In this review, we will provide a historical context of neural network research's involvement with complex systems, and highlight several active areas in modern deep learning research that incorporate the principles of collective intelligence to advance its current capabilities. To facilitate a bi-directional flow of ideas, we also discuss work that utilize modern deep learning models to help advance complex systems research. We hope this review can serve as a bridge between complex systems and deep learning communities to facilitate the cross pollination of ideas and foster new collaborations across disciplines.

show abstract

“…1) General MARL Framework: The development of MARL is relatively new compared to its single-agent counterpart, and there is currently no commonly and widely used MARL framework. PettingZoo [10], a Python library, has a goal of developing a universal application programming interface (API) for formulating MARL problems, as OpenAI Gym [11] did for single-agent RL problems. However, because the key advantages of PettingZoo-namely, an efficient formulation suitable for the turn-based games and an ability to handle agent creation and death within episodes-are less relevant to power system control problems, PowerGridworld does not adopt the PettingZoo APIs at this stage for simplicity.…”

Section: B Related Softwarementioning

confidence: 99%

PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems

Biagioni¹,

Zhang²,

Wald³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present the PowerGridworld software package to provide users with a lightweight, modular, and customizable framework for creating power-systems-focused, multi-agent Gym environments that readily integrate with existing training frameworks for reinforcement learning (RL). Although many frameworks exist for training multi-agent RL (MARL) policies, none can rapidly prototype and develop the environments themselves, especially in the context of heterogeneous (composite, multidevice) power systems where power flow solutions are required to define grid-level variables and costs. PowerGridworld is an opensource software package that helps to fill this gap. To highlight PowerGridworld's key features, we present two case studies and demonstrate learning MARL policies using both OpenAI's multi-agent deep deterministic policy gradient (MADDPG) and RLLib's proximal policy optimization (PPO) algorithms. In both cases, at least some subset of agents incorporates elements of the power flow solution at each time step as part of their reward (negative cost) structures.

show abstract

PettingZoo: Gym for Multi-Agent Reinforcement Learning

Cited by 33 publications

References 24 publications

Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees

Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees

Collective Intelligence for Deep Learning: A Survey of Recent Developments

PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems

Contact Info

Product

Resources

About