Learning and Exploiting Shaped Reward Models for Large Scale Multiagent RL

Abstract

Many real world systems involve interaction among large number of agents to achieve a common goal, for example, air traffic control. Several model-free RL algorithms have been proposed for such settings. A key limitation is that the empirical reward signal in model-free case is not very effective in addressing the multiagent credit assignment problem, which determines an agent's contribution to the team's success. This results in lower solution quality and high sample complexity. To address this, we contribute (a) an approach to learn a differentiable reward model for both continuous and discrete action setting by exploiting the collective nature of interactions among agents, a feature commonly present in large scale multiagent applications; (b) a shaped reward model analytically derived from the learned reward model to address the key challenge of credit assignment; (c) a model-based multiagent RL approach that integrates shaped rewards into well known RL algorithms such as policy gradient, soft-actor critic. Compared to previous methods, our learned reward models are more accurate, and our approaches achieve better solution quality on synthetic and real world instances of air traffic control, and cooperative navigation with large agent population.

Collaborative Traffic Signal Automation Using Deep Q-Learning

“…In order to achieve these objectives, each agent collaborates with neighboring nodes to increase its own reward and contribute to the rewards of others. Drawing inspiration from the principles of frequent measurability and spatial decomposability [57], we have defined a local reward function, r, for each intersection, which enables us to achieve these goals in a highly efficient and effective manner.…”