Character controllers using motion VAEs

Abstract

Fig. 1. Given example data, we learn an autoregressive conditional variational autoencoder (VAE) that predicts the next pose one frame at a time. A variety of task-specific control policies can then be learned on top of this model.A fundamental problem in computer animation is that of realizing purposeful and realistic human movement given a sufficiently-rich set of motion capture clips. We learn data-driven generative models of human movement using autoregressive conditional variational autoencoders, or Motion VAEs. The latent variables of the learned autoencoder define the action space for the movement and thereby govern its evolution over time. Planning or control algorithms can then use this action space to generate desired motions. In particular, we use deep reinforcement learning to learn controllers that achieve goal-directed movements. We demonstrate the effectiveness of the approach on multiple tasks. We further evaluate system-design choices and describe the current limitations of Motion VAEs.

MoGlow

“…VAEs circumvent computational issues by using a variational and amortised (see Cremer et al [2018]) approximation of the likelihood for training. They have been applied to model controllable human locomotion [Habibie et al 2017;Ling et al 2020] and to generate head motion from speech [Greenwood et al 2017a,b]. Ling et al [2020] describes an autogregressive unconditional motion model based on VAEs, using a deterministic decoder based on the mixture-of-experts architecture from .…”

“…They have been applied to model controllable human locomotion [Habibie et al 2017;Ling et al 2020] and to generate head motion from speech [Greenwood et al 2017a,b]. Ling et al [2020] describes an autogregressive unconditional motion model based on VAEs, using a deterministic decoder based on the mixture-of-experts architecture from . -VAEs [Higgins et al 2016] are used to mitigate posterior collapse, while scheduled sampling ] is necessary to stabilise long-term motion generation.…”

Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model

“…We attempt to mitigate these drawbacks in our proposed model by formulating the internal state and latent code in two separate channels and conditioning the latent code to the previous internal state to model the temporal dependencies during test time. Recently, [LZCVDP20] et al proposed an interesting model based on VAEs where the motion is controlled by setting the latent code as the output of a deep reinforcement learning module. Unlike our method, they modelled the motion by a Markovian assumption, meaning that each pose only depends on the previous pose and the autoregressive model is memoryless.…”

“…Mixture‐of‐Experts Approaches : Another strategy exploited in [SZKS19, SZKZ20, LZCVDP20] to address the problem of mean collapse in multi‐modal motion data is to use a Mixture‐of‐Experts (MoE) network where each expert is responsible for one mode in the training data. Though effective at mitigating mean collapse, the number of parameters in these networks increases with the number of experts.…”

“…l is considered as a hyperparameter where the best results were achieved for l = 3. The dimension of the Motion Word D w was set to 32 * 3 = 96 which is equal to the approximate degrees of freedom in each pose ([LZCVDP20, PALvdP18]) times the length of each subsequence. To model the recursive structure of Motion Words we used a variational recurrent neural network [CKD*15] extended to condition on the control parameters.…”