On Retrospecting Human Dynamics with Attention

Abstract

Deep recurrent neural networks have achieved impressive success in forecasting human motion with a sequence to sequence architecture. However, forecasting in longer time horizons often leads to implausible human poses or converges to mean poses, because of error accumulation and difficulties in keeping track of longerterm information. To address these challenges, we propose to retrospect human dynamics with attention. A retrospection module is designed upon RNN to regularly retrospect past frames and correct mistakes in time. This significantly improves the memory of RNN and provides sufficient information for the decoder networks to generate longer term prediction. Moreover, we present a spatial attention module to explore and exploit cooperation among joints in performing a particular motion. Residual connections are also included to guarantee the performance of short term prediction. We evaluate the proposed algorithm on the largest and most challenging Human 3.6M dataset in the field. Experimental results demonstrate the necessity of investigating motion prediction in a self audit manner and the effectiveness of the proposed algorithm in both short term and long term predictions.

A Quadruple Diffusion Convolutional Recurrent Network for Human Motion Prediction

“…In heuristic research for motion prediction, a representative residual network [8] was first proposed to estimate velocity, which has achieved great success in reducing initial discontinuity of the generated sequence compared with previous attempts [30], [31] predicting only static poses. This triggers many sequential-based motion prediction frameworks [13], [16], [32] introducing residual connection into their baselines. One step of residual connection means that the system outputs velocity from the pose, and adds the velocity back to the previous pose to predict the next step.…”

“…However, their learned temporal dependency is restricted by a deterministic filter size, causing an intensive long-term dynamic loss in prediction. Recently, Dong and Xu [32] attempted to reduce long-term error by looking back at previous frames with spatial attention. Chen et al [34] avoided motion drift by generating early prediction controlled by the action label, while our model is label-agnostic and is also feasible for long-term prediction.…”

“…Our approach follows the intention of multilabel algorithms. We then compare with the state-of-the-art multi-label algorithms related to our network architecture under two types of baselines, which are RNN or CNNbased models: RRNN [8], 3D-PFNet [10], RMA [32], TP-RNN [17], VGRU [10], QuaterNet [36], BiHMP-GAN [16], and ConvSeq2seq [15]; GCN-based models: LDR [29] and DMGNN [28]. To demonstrate the effectiveness of our velocity modeling method, we also present the prediction results for modeling velocity consistently (denoted as VRNN) to compare with RRNN which models posture sequence with residual connections.…”

“…Mean Angle Error (MAE) Following the standard evaluation protocol adopted in [8], [15], [16], [32], we first use the mean error of Euler angle as the evaluation metric for a fair comparison among the baselines and the proposed method. The prediction error is calculated from the average of Euler angle difference per joint between prediction and reference.…”

Spatial–Temporal Attention-Based Human Dynamics Retrospection

“…This work is an expanded version of conference publication [9]. Compared with previous version of this work, we make following extensions: 1) We enrich the content of abstract, introduction and related work to cover sufficient details.…”