GlocalNet: Class-aware Long-term Human Motion Synthesis

Battan, Neeraj; Agrawal, Yudhik; Rao, Sai Soorya; Goel, Aman; Sharma, Avinash

doi:10.1109/wacv48630.2021.00092

Cited by 13 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, the teacher policy is first learned to achieve the tasks in optimal ways and then the time-critical student policy is followed to improve the responsiveness of the interactive control. Battan et al [28] synthesize motion from the input label and give an initial set of frames using a 2-stage approach on an encoder-decoder architecture. In particular, the first stage predicts a sparse set of keyframes for the whole motion while the second stage generates the dense motion trajectories from the output of the first stage.…”

Section: Action-level Motion Synthesismentioning

confidence: 99%

Interaction Mix and Match: Synthesizing Close Interaction using Conditional Hierarchical GAN with Multi-Hot Class Embedding

Goel¹,

Men²,

Ho³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Synthesizing multi-character interactions is a challenging task due to the complex and varied interactions between the characters. In particular, precise spatiotemporal alignment between characters is required in generating close interactions such as dancing and fighting. Existing work in generating multi-character interactions focuses on generating a single type of reactive motion for a given sequence which results in a lack of variety of the resultant motions. In this paper, we propose a novel way to create realistic human reactive motions which are not presented in the given dataset by mixing and matching different types of close interactions. We propose a Conditional Hierarchical Generative Adversarial Network with Multi-Hot Class Embedding to generate the Mix and Match reactive motions of the follower from a given motion sequence of the leader. Experiments are conducted on both noisy (depth-based) and high-quality (MoCap-based) interaction datasets. The quantitative and qualitative results show that our approach outperforms the state-of-the-art methods on the given datasets. We also provide an augmented dataset with realistic reactive motions to stimulate future research in this area. The code is available at https://github.com/Aman-Goel1/IMM

show abstract

Section: Action-level Motion Synthesismentioning

confidence: 99%

Interaction Mix and Match: Synthesizing Close Interaction using Conditional Hierarchical GAN with Multi-Hot Class Embedding

Goel¹,

Men²,

Ho³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Among deep generative models, motion generation based on Recurrent Neural Network (RNN) becomes the mainstream with its effectiveness in creating sequential movements. With RNN backbones, [15] incorporated label information as guidance to synthesize desired future motions based on the initial given poses, and [14] retained spatial and temporal structural information in the generated motion using graph convolutional layers. Some researches [13,25] also adopt variational auto-encoder to learn a competitive motion manifold that can generate stylistic or long-term dynamics with stochastic patterns.…”

Section: Deep Generative Models In Motion Synthesismentioning

confidence: 99%

GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction

Men¹,

Shum²,

Ho³

et al. 2021

Preprint

View full text Add to dashboard Cite

Creating realistic characters that can react to the users' or another character's movement can benefit computer graphics, games and virtual reality hugely. However, synthesizing such reactive motions in human-human interactions is a challenging task due to the many different ways two humans can interact. While there are a number of successful researches in adapting the generative adversarial network (GAN) in synthesizing single human actions, there are very few on modelling human-human interactions. In this paper, we propose a semi-supervised GAN system that synthesizes the reactive motion of a character given the active motion from another character. Our key insights are two-fold. First, to effectively encode the complicated spatial-temporal information of a human motion, we empower the generator with a part-based long short-term memory (LSTM) module, such that the temporal movement of different limbs can be effectively modelled. We further include an attention module such that the temporal significance of the interaction can be learned, which enhances the temporal alignment of the active-reactive motion pair. Second, as the reactive motion of different types of interactions can be significantly different, we introduce a discriminator that not only tells if the generated movement is realistic or not, but also tells the class label of the interaction. This allows the use of such labels in supervising the training of the generator. We experiment with the SBU and the HHOI datasets. The high quality of the synthetic motion demonstrates the effective design of our generator, and the discriminability of the synthesis also demonstrates the strength of our discriminator.

show abstract

“…Action prediction: A small initial action sequence is used to condition the generation of the full version in action prediction task. One set of approaches employ a sequence-to-sequence paradigm to predict joints [6,11] or joint velocities [29]. Other approaches use adversarial generative models [5,24] and conditioned autoregressive models [19] to synthesize human leg motion.…”

Section: Related Workmentioning

confidence: 99%

“…However, they train separate models for each action category. To produce sequences, Battan et al [6] use a two stage approach involving sparse and dense motion prediction. In general, however, end-to-end action synthesis is more challenging due to the absence of input priming used for action prediction.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

MUGL: Large Scale Multi Person Conditional Action Generation with Locomotion

Maheshwari¹,

Gupta²,

Sarvadevabhatla³

2021

Preprint

View full text Add to dashboard Cite

Figure 1: Examples of single and multi-person human pose action sequences generated by our model, MUGL. The user can select from among a large number of action classes (120). The changing length of dotted line connecting kinematic tree root joints of individuals in two-person actions indicates change in their relative global positions. Note the distinction from single person actions where individual frames correspond to timesteps, i.e. global displacements are absent. Also note the variable length of sequences generated. Additional results in Fig. 3.

show abstract

GlocalNet: Class-aware Long-term Human Motion Synthesis

Cited by 13 publications

References 20 publications

Interaction Mix and Match: Synthesizing Close Interaction using Conditional Hierarchical GAN with Multi-Hot Class Embedding

Interaction Mix and Match: Synthesizing Close Interaction using Conditional Hierarchical GAN with Multi-Hot Class Embedding

GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction

MUGL: Large Scale Multi Person Conditional Action Generation with Locomotion

Contact Info

Product

Resources

About