Learning Agent Communication under Limited Bandwidth by Message Pruning

Abstract

Communication is a crucial factor for the big multi-agent world to stay organized and productive. Recently, Deep Reinforcement Learning (DRL) has been applied to learn the communication strategy and the control policy for multiple agents. However, the practical limited bandwidth in multiagent communication has been largely ignored by the existing DRL methods. Specifically, many methods keep sending messages incessantly, which consumes too much bandwidth. As a result, they are inapplicable to multi-agent systems with limited bandwidth. To handle this problem, we propose a gating mechanism to adaptively prune less beneficial messages. We evaluate the gating mechanism on several tasks. Experiments demonstrate that it can prune a lot of messages with little impact on performance. In fact, the performance may be greatly improved by pruning redundant messages. Moreover, the proposed gating mechanism is applicable to several previous methods, equipping them the ability to address bandwidth restricted settings.

Event-Triggered Communication Network With Limited-Bandwidth Constraint for Multi-Agent Reinforcement Learning

“…A predator only has a local view around itself, so they have to cooperate to capture preys and are required to avoid collision with other predators. Our baselines for limited bandwidth are (1) Gated-ACML (Mao et al 2019), (2) SchedNet , and (3) Message-Dropout (Kilinc and Montana 2018). In addition, A3C2 (Simões, Lau, and Reis 2020) is introduced as a full-communication version.…”

“…IC3Net (Singh, Jain, and Sukhbaatar 2018) extends the work of CommNet (Sukhbaatar, Fergus et al 2016) by means of Long Short-Term Memory (LSTM) and the gating mechanism. Gated-ACML (Mao et al 2019) and ATOC (Jiang and Lu 2018) both evaluate the importance of communication by comparing the Q-difference between sending messages and not. If the difference is greater than a threshold, agents consider the message is valuable and choose to communicate.…”

“…Beyond that, if no message is received, the agent uses the memorized message rather than the zero vector (e.g. Gated-ACML (Mao et al 2019), ATOC (Jiang and Lu 2018), and SchedNet )) to prevent the lose of information and preserve the cooperation performance.…”

“…If there are a large number of agents and they send messages excessively, the network can be easily blocked, delaying message transmission and impairing cooperative effects. Some research in MARL has been proposed to learn communication with limited bandwidth (Jiang and Lu 2018;Mao et al 2019;Wang et al 2019). But some key problems are still not well studied.…”

“…To verify the effectiveness, two typical multi-agent tasks, including cooperative navigation and predator-prey, are simulated. We compare our method with other MARL methods that also consider the limited-bandwidth constraint, including SchedNet , Gated-ACML (Mao et al 2019), and Message-dropout (Kim, Cho, and Sung 2019). After comparison, our ETCNet is significant in reducing the bandwidth consumption and preserves the whole system with marginal impact.…”