NeuPL: Neural Population Learning

Abstract

Learning in strategy games (e.g. StarCraft, poker) requires the discovery of diverse policies. This is often achieved by iteratively training new policies against existing ones, growing a policy population that is robust to exploit. This iterative approach suffers from two issues in real-world games: a) under finite budget, approximate best-response operators at each iteration needs truncating, resulting in under-trained good-responses populating the population; b) repeated learning of basic skills at each iteration is wasteful and becomes intractable in the presence of increasingly strong opponents. In this work, we propose Neural Population Learning (NeuPL) as a solution to both issues. NeuPL offers convergence guarantees to a population of best-responses under mild assumptions. By representing a population of policies within a single conditional model, NeuPL enables transfer learning across policies. Empirically, we show the generality, improved performance and efficiency of NeuPL across several test domains 1 . Most interestingly, we show that novel strategies become more accessible, not less, as the neural population expands. * Currently at Reality Labs, work carried out while at DeepMind. † Work carried out while at DeepMind. 1 See https://neupl.github.io/demo/ for supplementary illustrations. 2 This is formally quantified by Relative Population Performance, see Definition A.1 (Balduzzi et al., 2019).