Multi-lingual agents through multi-headed neural networks

Abstract

This paper considers cooperative Multi-Agent Reinforcement Learning, focusing on emergent communication in settings where multiple pairs of independent learners interact at varying frequencies. In this context, multiple distinct and incompatible languages can emerge. When an agent encounters a speaker of an alternative language, there is a requirement for a period of adaptation before they can efficiently converse. This adaptation results in the emergence of a new language and the forgetting of the previous language. In principle, this is an example of the Catastrophic Forgetting problem which can be mitigated by enabling the agents to learn and maintain multiple languages. We take inspiration from the Continual Learning literature and equip our agents with multi-headed neural networks which enable our agents to be multi-lingual. Our method is empirically validated within a referential MNIST based communication game and is shown to be able to maintain multiple languages where existing approaches cannot.This paper focuses on emergent communication within multi-agent reinforcement learning (MARL), specifically addressing settings where agents can be considered as independent learners (IL) [1]. This restriction removes common methodologies which are utilised to improve training speed and stability, such as centralised training decentralised execution (CDTE) [2], parameter sharing [1] and gradient propagation through other agents. This is justified by the motivation of creating algorithms that better approximate human learning, where, for example, models of other agents are unlikely to be available for gradient propagation. Recent work has attempted to improve training efficiency through a variety of methods. Jaques et al. [3] proposes an intrinsic reward based on social influence to encourage communication of useful information and Eccles et al. [4] proposes the introduction of biases to promote the emergence of communication.In this more natural setting, experimentation has generally been restricted to two independent agents. However, realistic scenarios are likely to involve larger numbers of independent agents interacting at varying frequencies. As the agents do not use parameter sharing, it is conceivable that multiple unique languages may arise where these languages are unlikely to be compatible. As result of this, any interaction with a new agent mandates the learning of a shared language. Without specific 35th Conference on Neural Information Processing Systems (NeurIPS 2021),