From motor control to team play in simulated humanoid football

Abstract

Learning to combine control at the level of joint torques with longer-term goal-directed behavior is a long-standing challenge for physically embodied artificial agents. Intelligent behavior in the physical world unfolds across multiple spatial and temporal scales: Although movements are ultimately executed at the level of instantaneous muscle tensions or joint torques, they must be selected to serve goals that are defined on much longer time scales and that often involve complex interactions with the environment and other agents. Recent research has demonstrated the potential of learning-based approaches applied to the respective problems of complex movement, long-term planning, and multiagent coordination. However, their integration traditionally required the design and optimization of independent subsystems and remains challenging. In this work, we tackled the integration of motor control and long-horizon decision-making in the context of simulated humanoid football, which requires agile motor control and multiagent coordination. We optimized teams of agents to play simulated football via reinforcement learning, constraining the solution space to that of plausible movements learned using human motion capture data. They were trained to maximize several environment rewards and to imitate pretrained football-specific skills if doing so led to improved performance. The result is a team of coordinated humanoid football players that exhibit complex behavior at different scales, quantified by a range of analysis and statistics, including those used in real-world sport analytics. Our work constitutes a complete demonstration of learned integrated decision-making at multiple scales in a multiagent setting.

Human Motion Synthesis Using Trigonometric Splines

“…A human dynamic model with 48 DOF was used to visualize and simulate human motion based on experimental motion capture data [30]. Multi-character simulations were also obtained from motion capture data using reinforcement learning [31], [32] and hierarchical reinforcement learning [33]. Data-driven human motion simulations are popular in computer graphics, gaming, and the machine learning community, which can be explained in part by the presence of model-free deep reinforcement learning techniques [19].…”

Price of Anarchy in a Double-Sided Critical Distribution System

“…In this section, we describe a reinforced learning algorithm we use to obtain an approximation of the equilibrium of the Crisis. We treat the entire interaction as a multi-agent reinforcement learning (MARL) problem as it is common in the literature [6,10,12,13], with the assumption that the learning algorithm shall converge to a solution close to the equilibrium. We further verify the quality of the solution by computing its exploitability [9].…”

Behavioural cloning based RL agents for district energy management

“…Pezzotti et al developed MimicBot, a deep RL agent in a Fantasy Football game environment, which achieved excellent performance when RL training techniques were applied after an initial behaviour cloning step [9]. In a more advanced scenario, Liu et al created a framework to train simulated humanoid figures utilizing motion-capture data, and showed that complex behaviours requiring multi-agent coordination could be captured using this approach [4].…”