MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics

Yan, Xinchen; Rastogi, Akash; Villegas, Ruben; Sunkavalli, Kalyan; Shechtman, Eli; Hadap, Sunil; Yumer, Ersin; Lee, Honglak

doi:10.1007/978-3-030-01228-1_17

Cited by 111 publications

(102 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It enables realistic joint rotations transformation across subjects through cycle consistency based adversarial training, and then the subject-specific motion is generated by the prior-known basic skeleton and the kinematics. Yan et al [38] adopted a variational recurrent auto-encoder to perform the multi-modal motion transformation from one to another. The embedding feature was generated by the variational auto-encoder and it was further enhanced by motion retargeting.…”

Section: B Cross-subject Human Motion Retargetingmentioning

confidence: 99%

Cross-Subject and Cross-Modal Transfer for Generalized Abnormal Gait Pattern Recognition

Guo

Deligianni

et al. 2021

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

For abnormal gait recognition, pattern-specific features indicating abnormalities are interleaved with the subject-specific differences representing biometric traits. Deep representations are therefore prone to overfitting and the models derived cannot generalize well to new subjects. Furthermore, there is limited availability of abnormal gait data obtained from precise Motion Capture (Mocap) systems because of regulatory issues and slow adaptation of new technologies in health care. On the other hand, data captured from markerless vision sensors or wearable sensors can be obtained in home environments but noises from such devices may prevent effective extraction of relevant features. To address these challenges, we propose a cascade of deep architectures that can encode cross-modal and cross-subject transfer for abnormal gait recognition. Cross-modal transfer maps noisy data obtained from RGBD and wearable sensors to accurate four-dimensional (4D) representations of the lower limb and joints obtained from the Mocap system. Subsequently, cross-subject transfer allows to disentangle subject-specific from abnormal pattern-specific gait features based on a multi-encoder autoencoder architecture. To validate the proposed methodology, we obtained multi-modal gait data based on a multi-camera motion capture system along with synchronized recordings of Electromyography (EMG) data and 4D skeleton data extracted from a single RGBD camera. Classification accuracy was improved significantly in both Mocap and noisy modalities.

show abstract

Section: B Cross-subject Human Motion Retargetingmentioning

confidence: 99%

Cross-Subject and Cross-Modal Transfer for Generalized Abnormal Gait Pattern Recognition

Guo

Deligianni

et al. 2021

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…RNNs have been extensively explored for short and longterm human motion prediction in computer vision [Fragkiadaki et al 2015;Martinez et al 2017]. It is commonly noted that these models can often be unstable for long-term sequence prediction, and the production of long-term stable sequences is considered an accomplishment, even in the absence of a control task [Habibie et al 2017;Yan et al 2018;Zhou et al 2018].…”

Section: Kinematic Motion Synthesismentioning

confidence: 99%

Character controllers using motion VAEs

et al. 2020

View full text Add to dashboard Cite

Fig. 1. Given example data, we learn an autoregressive conditional variational autoencoder (VAE) that predicts the next pose one frame at a time. A variety of task-specific control policies can then be learned on top of this model.A fundamental problem in computer animation is that of realizing purposeful and realistic human movement given a sufficiently-rich set of motion capture clips. We learn data-driven generative models of human movement using autoregressive conditional variational autoencoders, or Motion VAEs. The latent variables of the learned autoencoder define the action space for the movement and thereby govern its evolution over time. Planning or control algorithms can then use this action space to generate desired motions. In particular, we use deep reinforcement learning to learn controllers that achieve goal-directed movements. We demonstrate the effectiveness of the approach on multiple tasks. We further evaluate system-design choices and describe the current limitations of Motion VAEs.

show abstract

“…In [3], a pose generator and motion generator are trained progressively with the help of GANs for human 2D motion generation. Other methods [13,36] directly generate 2D human motions by VAEs or GANs. However, direct modeling of motions in 2D is inherently insufficient to capture the underlying 3D human shape articulations.…”

Section: Introductionmentioning

confidence: 99%

“…However, direct modeling of motions in 2D is inherently insufficient to capture the underlying 3D human shape articulations. Existing methods [3,18,36,37] often approach human poses in terms of the joints' coordinate locations, which unnecessarily entangle the human skeletons and their motion trajectories, and introduce extra barriers in faithful modeling of human kinematics. To further complicate the matter, the variation of generated human dynamics could be severely limited by the initial priors [37].…”

Section: Introductionmentioning

confidence: 99%