Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Abstract

We study the problem of training a Reinforcement Learning (RL) agent that is collaborative with humans without using human data. Although such agents can be obtained through self-play training, they can suffer significantly from the distributional shift when paired with unencountered partners, such as humans. In this paper, we propose Maximum Entropy Population-based training (MEP) to mitigate such distributional shift. In MEP, agents in the population are trained with our derived Population Entropy bonus to promote the pairwise diversity between agents and the individual diversity of agents themselves. After obtaining this diversified population, a common best agent is trained by paring with agents in this population via prioritized sampling, where the prioritization is dynamically adjusted based on the training progress. We demonstrate the effectiveness of our method MEP, with comparison to Self-Play PPO (SP), Population-Based Training (PBT), Trajectory Diversity (TrajeDi), and Fictitious Co-Play (FCP) in both matrix game and Overcooked game environments, with partners being human proxy models and real humans. A supplementary video showing experimental results is available at https://youtu.be/Xh-FKD0AAKE.

ProAgent: Building Proactive Cooperative Agents with Large Language Models

“…However, these methods may still limit the agent's cooperation ability in familiar tasks and fail to handle unseen tasks or new agent interactions. Another line of research focuses on zero-shot coordination (ZSC), utilizing Population-Based Training (PBT;Strouse et al 2021;Zhao et al 2023;Lupu et al 2021;Lucas and Allen 2022;Li et al 2023bLi et al , 2024 and Theory of Mind (ToM; Hu et al 2021a;Wu et al 2021;Wang et al 2021) to facilitate adaptive policy development for coordinating with various counterparts without prior coordination experience. However, these ZSC methods demand significant computational resources for data collection and model optimization, and the resulting policies often lack interpretability.…”

“…In previous works on Overcooked-AI, the cooperative performance of an agent is often evaluated with two held-out populations: self-play (SP) agent and human proxy model. We conduct a comparative analysis between our proposed ProAgent and five alternatives prevalent in the field including SP (Tesauro 1994;Carroll et al 2019), PBT (Jaderberg et al 2017), FCP (Strouse et al 2021), MEP (Zhao et al 2023), and COLE (Li et al 2023b(Li et al , 2024. We combined the above six algorithms in pairs to construct 36 pairs.…”

“…This research delves into the capabilities of LLMs in tackling the intricate challenges of multi-agent coordination (Yang and Wang 2021;Zhang, Yang, and Bas ¸ar 2021;Gronauer and Diepold 2022), particularly in the realm of policy generalization (Strouse et al 2021;Zhao et al 2023;Li et al 2023bLi et al , 2024. Current approaches (Carroll et al 2019;Jaderberg et al 2017;Strouse et al 2021;Zhao et al 2023;Li et al 2023bLi et al , 2024 to developing cooperative agents rely primarily on learning-based methods, whose policy generalization depends heavily on the diversity of teammates they interact with during the training phase. Such reliance, however, constrains the agents' capacity for strategic adaptation when cooperating with unfamiliar teammates, which becomes a significant challenge in zero-shot coordination scenarios.…”