Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model

Ghorbani, Saeed; Wloka, Calden; Etemad, Ali; Brubaker, Marcus A.; Troje, Nikolaus F.

doi:10.1111/cgf.14116

Cited by 20 publications

(17 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several authors [HHS*17, GWE*20] ascribed the regression towards a static pose to the ill‐posedness of long‐term motion prediction and advocated the use of external control signals to disambiguate the task. For instance, Ghorbani et al.…”

Section: Motion Synthesismentioning

confidence: 99%

A Survey on Deep Learning for Skeleton‐Based Human Animation

Mourot

Hoyet

Clerc³

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

Human character animation is often critical in entertainment content production, including video games, virtual reality or fiction films. To this end, deep neural networks drive most recent advances through deep learning (DL) and deep reinforcement learning (DRL). In this article, we propose a comprehensive survey on the state‐of‐the‐art approaches based on either DL or DRL in skeleton‐based human character animation. First, we introduce motion data representations, most common human motion datasets and how basic deep models can be enhanced to foster learning of spatial and temporal patterns in motion data. Second, we cover state‐of‐the‐art approaches divided into three large families of applications in human animation pipelines: motion synthesis, character control and motion editing. Finally, we discuss the limitations of the current state‐of‐the‐art methods based on DL and/or DRL in skeletal human character animation and possible directions of future research to alleviate current limitations and meet animators' needs.

show abstract

Section: Motion Synthesismentioning

confidence: 99%

A Survey on Deep Learning for Skeleton‐Based Human Animation

Mourot

Hoyet

Clerc³

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…The large number of 90 individual actors who performed the same set of actions, provides high diversity across performers in terms of action type, action execution, style, and modalities (video, motion capture, and IMU) which are important factors for research on action recognition (see e.g., [ 29 ] who used our dataset for action recognition). This is also important for frameworks for designing character animation that focus on modelling the natural stochasticity and diversity of the movements (e.g, [ 5 ]).…”

Section: Applicationsmentioning

confidence: 99%

“…Capturing, modelling, and simulating human body shape and kinematics has been an area of intense study in the fields of biomechanics, computer vision, and computer graphics, with applications including human-machine interactions [1], assistive healthcare [2], clinical diagnostics [3], and realistic computer animation pipelines [4][5][6]. In order to obtain body pose and kinematics at a resolution that is fine enough to make inferences about identity, action, and particularly stylistic features, we need large, high-quality datasets that can be used in both generative and discriminative contexts.…”

Section: Introductionmentioning

confidence: 99%

MoVi: A large multi-purpose human motion and video dataset

et al. 2021

Self Cite

View full text Add to dashboard Cite

Large high-quality datasets of human body shape and kinematics lay the foundation for modelling and simulation approaches in computer vision, computer graphics, and biomechanics. Creating datasets that combine naturalistic recordings with high-accuracy data about ground truth body shape and pose is challenging because different motion recording systems are either optimized for one or the other. We address this issue in our dataset by using different hardware systems to record partially overlapping information and synchronized data that lend themselves to transfer learning. This multimodal dataset contains 9 hours of optical motion capture data, 17 hours of video data from 4 different points of view recorded by stationary and hand-held cameras, and 6.6 hours of inertial measurement units data recorded from 60 female and 30 male actors performing a collection of 21 everyday actions and sports movements. The processed motion capture data is also available as realistic 3D human meshes. We anticipate use of this dataset for research on human pose estimation, action recognition, motion modelling, gait analysis, and body shape reconstruction.

show abstract

“…Habibie et al [8] proposed a VAE with a recurrent structure for controlled motion synthesis. In [14], a hierarchical recurrent model is proposed with each motion sub-sequence mapped to a stochastic latent code through a VAE. VAE-based methods have also found applications in cross-modal synthesis, generating motion from speech [15].…”

Section: A Deep Learning For Motion Prediction and Generationmentioning

confidence: 99%

Graph-based Normalizing Flow for Human Motion Generation and Reconstruction

Yin

Kragić

et al. 2021

Preprint

View full text Add to dashboard Cite

Data-driven approaches for modeling human skeletal motion have found various applications in interactive media and social robotics. Challenges remain in these fields for generating high-fidelity samples and robustly reconstructing motion from imperfect input data, due to e.g. missed marker detection. In this paper, we propose a probabilistic generative model to synthesize and reconstruct long horizon motion sequences conditioned on past information and control signals, such as the path along which an individual is moving. Our method adapts the existing work MoGlow by introducing a new graph-based model. The model leverages the spatial-temporal graph convolutional network (ST-GCN) to effectively capture the spatial structure and temporal correlation of skeletal motion data at multiple scales. We evaluate the models on a mixture of motion capture datasets of human locomotion with foot-step and bone-length analysis. The results demonstrate the advantages of our model in reconstructing missing markers and achieving comparable results on generating realistic future poses. When the inputs are imperfect, our model shows improvements on robustness of generation.

show abstract

Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model

Cited by 20 publications

References 37 publications

A Survey on Deep Learning for Skeleton‐Based Human Animation

A Survey on Deep Learning for Skeleton‐Based Human Animation

MoVi: A large multi-purpose human motion and video dataset

Graph-based Normalizing Flow for Human Motion Generation and Reconstruction

Contact Info

Product

Resources

About