Learning and Exploring Motor Skills with Spacetime Bounds

Abstract

Learning cartwheels with spacetime bounds. The top green motion shows the reference, and the bottom yellow motions are simulations. The curves represent the Y position of the character's center of mass, and are colored to represent the reference (green), the simulations (yellow), and the spacetime bounds (red). The blue region illustrates the nonuniform feasible region under the given spacetime bounds. During training, episodes are terminated immediately once any spacetime bounds are violated, as shown in the bottom simulation.

Discovering diverse athletic jumping strategies

“…We explore a chosen lowdimensional feature space (3 ∼ 4𝐷) of the take-off states for learning diverse jumping strategies. As shown by previous work [Ma et al 2021], the take-off moment is a critical point of jumping motions, where the volume of the feasible region of the dynamic skill is the smallest. In another word, bad initial states will fail fast, which in a way help our exploration framework to find good ones quicker.…”

“…With the wide availability of motion capture data, many research endeavors have been focused on tracking-based controllers, which are capable of reproducing high-quality motions by imitating motion examples. Controllers for a wide range of skills have been demonstrated through trajectory optimization [da Silva et al 2008;Lee et al 2010Lee et al , 2014Muico et al 2009;Sok et al 2007;Ye and Liu 2010a], sampling-based algorithms [Liu et al 2016[Liu et al , 2015[Liu et al , 2010, and deep reinforcement learning [Liu and Hodgins 2018;Ma et al 2021;Peng et al 2018aPeng et al , 2017Peng et al , 2018bSeunghwan Lee and Lee 2019]. Tracking controllers have also been combined with kinematic motion generators to support interactive control of simulated characters [Bergamin et al 2019;Won et al 2020].…”

“…Even though trackingbased methods have demonstrated their effectiveness on achieving task-related goals [Peng et al 2018a], the imitation objective inherently restricts them from generalizing to novel motion strategies fundamentally different from the reference. Most recently, style exploration has also been demonstrated within a physics-based DRL framework using spacetime bounds [Ma et al 2021]. However, these remain style variations rather than strategy variations.…”

Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante

“…Secondly, they reduce the search space to sensible solutions only. Some local minima of poor recovery strategies are strongly discouraged from the start, which leads to more stable learning and faster convergence [38]. Numerical values are robot-specific.…”

“…The user is only supposed to rectify it occasionally and the operational space is likely to be confined in practice. We restrict the odometry displacement over a time window ∆q x,y,ϕ instead of its instantaneous velocity [38]. It limits the drift while allowing a few recovery steps.…”