From Motor Control to Team Play in Simulated Humanoid Football

Abstract

Intelligent behaviour in the physical world exhibits structure at multiple spatial and temporal scales. Although movements are ultimately executed at the level of instantaneous muscle tensions or joint torques, they must be selected so as to serve goals defined on much longer timescales, and in terms of relations that extend far beyond the body itself, ultimately involving coordination with other agents. Recent research in artificial intelligence has shown the promise of learning-based approaches to the respective problems of complex movement, longer-term planning, and multi-agent coordination. However, there is limited research aimed at their integration. We study this problem by training teams of physically simulated humanoid avatars to play football in a realistic virtual environment. We develop a method that combines imitation learning, single-and multi-agent reinforcement learning and population-based training, and makes use of transferable representations of behaviour for decision making at different levels of abstraction. In a sequence of training stages, players first learn to control a fully articulated body to perform realistic, human-like movements such as running and turning; they then acquire mid-level football skills such as dribbling and shooting; finally, they develop awareness of others and learn to play as a team, successfully bridging the gap between low-level motor control at a time scale of milliseconds, and coordinated goal-directed behaviour as a team at the timescale of tens of seconds. We investigate the emergence of behaviours at different levels of abstraction, as well as the representations that underlie these behaviours using several analysis techniques, including statistics from real-world sports analytics. Our work constitutes a complete demonstration of integrated decision-making at multiple scales in a physically embodied multi-agent setting. We provide footage of the learned football skills in the supplementary video. 1

TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations

“…Many researchers have developed agents for other football games (e.g., RoboCup Soccer Simulator (Kitano et al, 1997) and the DeepMind MuJoCo Multi-Agent Soccer Environment (Liu et al, 2019(Liu et al, , 2021a). Different from Google Research Football, these environments focus more on low-level control of a physics simulation of robots, while GRF focuses on high-level actions.…”

Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors

“…In particular it will be interesting to explore tasks that include more dynamic movements and obstacle traversal, or advanced object interaction [8,9]. Hierarchical controllers like in this work could also be particularly suitable for enabling more efficient learning of perceptive policies as has already been shown in simulation [8].…”

“…These methods can amortize the cost of complex optimization strategies at training time into neural-network (and other) controllers that are cheap to evaluate during deployment, and they can find flexible solutions that generalize to novel situations. A number of studies in simulation have demonstrated how Reinforcement Learning (RL) techniques can be deployed to generate complex movement strategies including perception-action coupling and object interaction [6,7,8,9]. Since learning directly on the hardware raises concerns related to data efficiency and safety (e.g.…”

“…Our work is inspired by ideas developed in the literature on character animation where human and animal Motion Capture (MoCap) data has been used extensively to enable controllable behaviors with smooth transitions between different movements, both via kinematic animation [31,32] and via controllers used in physical simulation models [7,33,34,35]. Going a step further, previous works such as [36] or [37,38,39] have investigated strategies to use MoCap data as a general purpose prior to constrain movements of downstream RL tasks, including complex whole body control and object interaction [8,9].…”

Learning to Advise and Learning from Advice in Cooperative Multi-Agent Reinforcement Learning

“…People are intrigued by creating collective artificial intelligence. Deep multi-agent reinforcement learning (MARL) is one of the inspiring achievements in cooperative agents [1]- [5] in recent years. However, learning to coordinate is still a daunting problem in MARL.…”