Scalable Multi-Agent Inverse Reinforcement Learning via Actor-Attention-Critic

Abstract

Multi-agent adversarial inverse reinforcement learning (MA-AIRL) is a recent approach that applies single-agent AIRL to multi-agent problems where we seek to recover both policies for our agents and reward functions that promote expertlike behavior. While MA-AIRL has promising results on cooperative and competitive tasks, it is sample-inefficient and has only been validated empirically for small numbers of agents -its ability to scale to many agents remains an open question. We propose a multi-agent inverse RL algorithm that is more sample-efficient and scalable than previous works. Specifically, we employ multi-agent actor-attention-critic (MAAC) -an off-policy multi-agent RL (MARL) method -for the RL inner loop of the inverse RL procedure. In doing so, we are able to increase sample efficiency compared to state-of-the-art baselines, across both small-and large-scale tasks. Moreover, the RL agents trained on the rewards recovered by our method better match the experts than those trained on the rewards derived from the baselines. Finally, our method requires far fewer agent-environment interactions, particularly as the number of agents increases.

Agent-Centric Representations for Multi-Agent Reinforcement Learning

“…Each agent receives a state input s i , covering the global s and a specification of the agent's location, and generates its policy π i = π(s i ) and state value V i = V (s i ). The model is shared between agents, adding scalability when the number of agents grows larger and the environment more complex (Iqbal and Sha, 2018;Jiang and Lu, 2018;Jeon et al, 2020). It also potentially alleviates instability due to the non-stationary nature of multi-agent environments by sharing the same embedding space (Lowe et al, 2017).…”