Convolutional Sequence to Sequence Model for Human Dynamics

Abstract

Human motion modeling is a classic problem in computer vision and graphics. Challenges in modeling human motion include high dimensional prediction as well as extremely complicated dynamics.We present a novel approach to human motion modeling based on convolutional neural networks (CNN). The hierarchical structure of CNN makes it capable of capturing both spatial and temporal correlations effectively. In our proposed approach, a convolutional long-term encoder is used to encode the whole given motion sequence into a long-term hidden variable, which is used with a decoder to predict the remainder of the sequence. The decoder itself also has an encoder-decoder structure, in which the short-term encoder encodes a shorter sequence to a short-term hidden variable, and the spatial decoder maps the long and short-term hidden variable to motion predictions. By using such a model, we are able to capture both invariant and dynamic information of human motion, which results in more accurate predictions. Experiments show that our algorithm outperforms the state-of-the-art methods on the Human3.6M and CMU Motion Capture datasets. Our code is available at the project website 1 . * Considered as equal contribution. 1 https://github.com/chaneyddtt/Convolutional-Sequence-to-Sequence-Model-for-Human-Dynamics

Learning Trajectory Dependencies for Human Motion Prediction

“…The left frames correspond to the observations. From top to bottom, we show the ground truth, and predictions obtained by the methods of [17] and [16], and by our approach on joint angles and 3d coordinates. Our predictions better match the ground truth.…”

“…Because of the temporal nature of the signal of interest, the most common trend consists of using Recurrent Neural Networks (RNNs) [7,11,17,9]. However, as argued in [9,16] , besides their well-known training difficulty [19], RNNs for motion prediction suffer from several drawbacks: First, existing works [7,17] that use the estimation at the current RNN step as input to the next prediction tend to accumulate errors throughout the generated sequence, leading to unrealistic predictions at inference time. Second, as observed in [16,17], earlier RNN-based methods [7,11] often produce strong discontinuities between the last observed frame and the first predicted one.…”

“…These discontinuities are partially due to the frame-by-frame regression procedure that does not encourage global smoothness of the sequence [9]. As a consequence, several works have proposed to rely on feed-forward networks for motion prediction [5,16]. In this paper, we introduce a new feed-forward approach to motion prediction, leading to more accurate predictions than RNN ones, as illustrated in Fig.…”

Unsupervised and Generic Short-Term Anticipation of Human Body Motions

“…• What is the role of incorporating delay inputs? Many neural network architectures incorporate delays [16][17][18], recurrent connections [2,19] or temporal convolutions [5,20,21], but the contribution of the delays or the memory cannot be separated from the overall network architecture.…”

Predicting the Future: A Jointly Learnt Model for Action Anticipation

“…Although there exist individual GAN models for anticipation [32,56], we take a step further in this work. The main contribution is the joint learning of a context descriptor for two tasks, action anticipation and representation prediction, through the joint training of two GANs.…”