Action-Agnostic Human Pose Forecasting

Abstract

Predicting and forecasting human dynamics is a very interesting but challenging task with several prospective applications in robotics, health-care, etc. Recently, several methods have been developed for human pose forecasting; however, they often introduce a number of limitations in their settings. For instance, previous work either focused only on short-term or long-term predictions, while sacrificing one or the other. Furthermore, they included the activity labels as part of the training process, and require them at testing time. These limitations confine the usage of pose forecasting models for real-world applications, as often there are no activity-related annotations for testing scenarios. In this paper, we propose a new action-agnostic method for shortand long-term human pose forecasting. To this end, we propose a new recurrent neural network for modeling the hierarchical and multi-scale characteristics of the human dynamics, denoted by triangular-prism RNN (TP-RNN). Our model captures the latent hierarchical structure embedded in temporal human pose sequences by encoding the temporal dependencies with different time-scales. For evaluation, we run an extensive set of experiments on Human 3.6M and Penn Action datasets and show that our method outperforms baseline and state-of-the-art methods quantitatively and qualitatively. Codes are available at https

A Quadruple Diffusion Convolutional Recurrent Network for Human Motion Prediction

“…5. As in [17], [52], the dataset is split into 1258 samples for training and 1068 for testing. Following [17], we input the initial velocity and predict the next 16 frames of poses.…”

“…Holden et al [2], [37] learned latent feature representations by operating 3D joint positions, which benefits multiple application fields like motion generation, recovery, and comparison. While training on 3D position suffers from skeleton constraints such as bone stretching, in [17] and [21], they modeled joint angles and tested on both angle and position spaces of their generations for a more comprehensive evaluation under different parameterizations. Following their work, for Human3.6M [38] and CMU MoCap datasets [39] we train on joint angles as they are invariant of bone length constraints and thus stabilizing the model fitting.…”

“…The "reconstruction loss" denotes the mean squared error between the ground-truth poses with the generated pose sequence. Since similar velocity chains can derive completely different pose sequences, it is risky to optimize the predictor on the velocity domain [17] when the network is blind to the generated poses. Therefore, we rebuild the poses from the predicted velocity frame by frame, and minimize the loss in the pose level, so that the generated motion is controllable.…”

“…Our approach follows the intention of multilabel algorithms. We then compare with the state-of-the-art multi-label algorithms related to our network architecture under two types of baselines, which are RNN or CNNbased models: RRNN [8], 3D-PFNet [10], RMA [32], TP-RNN [17], VGRU [10], QuaterNet [36], BiHMP-GAN [16], and ConvSeq2seq [15]; GCN-based models: LDR [29] and DMGNN [28]. To demonstrate the effectiveness of our velocity modeling method, we also present the prediction results for modeling velocity consistently (denoted as VRNN) to compare with RRNN which models posture sequence with residual connections.…”

“…To preserve the motion trend in long-term prediction, we propose a bi-directional predictor enhanced by a bidiscriminator to adversarially revise the generated forward and backward motion dynamics. From a single forward predictor [8], [10], [17], prediction errors are rapidly accumulated along the temporal domain since RNN models fail to keep the long-term knowledge in recurrent steps, causing the generated motion drifting to a wrong direction. To this end, we train a backward predictor to encode the velocity in reversed timesteps, such that the model recovers the context from the beginning dynamics that are lost during long sequence transition.…”