ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning

Abstract

Multi-agent reinforcement learning (MARL) has achieved promising results in recent years. However, most existing reinforcement learning methods require a large amount of data for model training. In addition, data-efficient reinforcement learning requires the construction of strong inductive biases, which are ignored in the current MARL approaches. Inspired by the symmetry phenomenon in multi-agent systems, this paper proposes a framework for exploiting prior knowledge by integrating data augmentation and a well-designed consistency loss into the existing MARL methods. In addition, the proposed framework is model-agnostic and can be applied to most of the current MARL algorithms. Experimental tests on multiple challenging tasks demonstrate the effectiveness of the proposed framework. Moreover, the proposed framework is applied to a physical multi-robot testbed to show its superiority.

Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning

“…In multi-agent systems, the symmetries are commonly referred to as equivariance and invariance (Yu et al 2023). Given a transformation operator L g : X → X and a mapping function f : X → Y, if there exists a second transformation operator K g : Y → Y in the output space of f such that:…”

“…For instance, again in Figure 1, multiple agents attempt to approach a target point where each agent can sense the environment, including information about other agents, obstacles, and the target point. Such problems, conditioned on the perfect symmetry transition function and symmetry reward function, are defined as symmetric Markov game in (van der Pol et al 2021;Yu et al 2023). Unfortunately, in the real world, there might exist imperfections in the environment, e.g., uneven ground, wind, and other non-uniform fields acting on the agents.…”