Neural animation layering for synthesizing martial arts movements

Abstract

Interactively synthesizing novel combinations and variations of character movements from different motion skills is a key problem in computer animation. In this paper, we propose a deep learning framework to produce a large variety of martial arts movements in a controllable manner from raw motion capture data. Our method imitates animation layering using neural networks with the aim to overcome typical challenges when mixing, blending and editing movements from unaligned motion sources. The framework can synthesize novel movements from given reference motions and simple user controls, and generate unseen sequences of locomotion, punching, kicking, avoiding and combinations thereof, but also reconstruct signature motions of different fighters, as well as close-character interactions such as clinching and carrying by learning the spatial joint relationships. To achieve this goal, we adopt a modular framework which is composed of the motion generator and a set of different control modules. The motion generator functions as a motion manifold that projects novel mixed/edited trajectories to natural full-body motions, and synthesizes realistic transitions between different motions. The control modules are task dependent and can be developed and trained separately by engineers to include novel motion tasks, which greatly reduces network iteration time when working with large-scale datasets. Our modular framework provides a transparent control interface for animators that allows modifying or combining movements after network training, and enables iterative adding of control modules for different motion tasks and behaviors. Our system can be used for offline and online motion generation alike, and is relevant for real-time applications such as computer games.

Spatio-Temporal Gating-Adjacency GCN for Human Motion Prediction

“…Mixture of Expert(MoE) [13,15] is a traditional machine learning method that uses blending coefficients generated by a gating network to blend multiple experts. For human motion, the gating network acts as a motion classifier to automatically calculate the probability that the input motion belongs to each class of motion and blends the results of the relevant experts to obtain the optimal output, which greatly improves the generalization of multiple human motion models [21,[29][30][31]38].…”

“…The Mixture of Experts(MoE) is a classic machine learning method, which is proven to be able to enhance the generalization of human motion models [21,[29][30][31]38]. The gating network is regarded as a motion classifier to automatically calculates the probability of which motion class the input motion belongs to.…”

GANimator: Neural Motion Synthesis from a Single Sequence

“…Henter et al [2020] propose another generative model for motion based on normalizing flow. Neural networks succeed in a variety of motion generation tasks: motion retargeting [Aberman et al 2020a[Aberman et al , 2019Villegas et al 2018], motion style transfer [Aberman et al 2020b;Mason et al 2022], key-frame based motion generation [Harvey et al 2020], motion matching [Holden et al 2020] and animation layering [Starke et al 2021]. It is worth noting that the success of deep learning methods hinges upon large and comprehensive mocap datasets.…”

Human Motion Control of Quadrupedal Robots using Deep Reinforcement Learning

“…Data-driven motion controllers have been proven effective for generating a wide range of physically plausible motions by leveraging motion capture data. Although kinematic approaches can provide interactive motion control [24,5,63,25,64,65], they cannot be directly transferred to real-world due to the lack of physical plausibility. On the other hand, physics-based motion trackers [42,43] allow us to obtain natural motions in simulation, but its control design requires additional manual efforts, such as feature selection and motion processing.…”