A Deep Bi-directional Attention Network for Human Motion Recovery

Abstract

Human motion capture (mocap) data, recording the movement of markers attached to specific joints, has gradually become the most popular solution of animation production. However, the raw motion data are often corrupted due to joint occlusion, marker shedding and the lack of equipment precision, which severely limits the performance in real-world applications. Since human motion is essentially a sequential data, the latest methods resort to variants of long short-time memory network (LSTM) to solve related problems, but most of them tend to obtain visually unreasonable results. This is mainly because these methods hardly capture long-term dependencies and cannot explicitly utilize relevant context, especially in long sequences. To address these issues, we propose a deep bi-directional attention network (BAN) which can not only capture the long-term dependencies but also adaptively extract relevant information at each time step. Moreover, the proposed model, embedded attention mechanism in the bi-directional LSTM (BLSTM) structure at the encoding and decoding stages, can decide where to borrow information and use it to recover corrupted frame effectively. Extensive experiments on CMU database demonstrate that the proposed model consistently outperforms other state-of-the-art methods in terms of recovery accuracy and visualization.

Deep Human Dynamics Prior

“…They are designed for specific problems and usually cannot solve other tasks without retraining mapping functions even if the problems are similar. For example, training a network for random joint missing samples is not easily re-applied to continuous joint corruption scenarios or to solve motion gaps [10]. Deep learning-based approaches typically generate the trained model from the training database.…”

“…Nowadays, Deep Neural Networks (DNNs) have been widely exploited in various tasks related to human motion [3,7,26]. They are committed to training an optimal model from vast mount training samples, and then use this trained model to repair damaged sequences in testing time [10]. Despite remarkable achievements, however, training data can hardly cover all action types due to the non-enumerability of human motion.…”

A Perceptual-Based Noise-Agnostic 3D Skeleton Motion Data Refinement Network

“…Mall et al [11] trained a set of filters using a deep, bidirectional, recurrent framework for clean, noisy and incomplete mocap data. In 2019, Cui et al [44] proposed a bidirectional attention network for missing data recovery, and their embedded attention mechanism can decide where to borrow information from and use this information to recover corrupted frames. The above deep-learning-based methods are action agnostic but noise specific, i.e., these methods can be trained by largescale data with a specified type of noise (such as Gaussian noise or missing data) and a heterogeneous mix of action types, and the network can refine any action with that noise type.…”

A variational U‐Net for motion retargeting

“…Tang et al 17 introduced a RNN‐based motion prediction system by analyzing the observed motion sequences. Cui et al 7 proposed a bidirectional RNN with an attention mechanism to accurately infer the missing joints. In contrast to ours, unsupervised motion retargeting systems have been proposed in References 18,19.…”

“…Against this backdrop, based on deep learning frameworks that can efficiently handle large amounts of motion data, 5‐7 here, we propose a novel deep autoencoder that combines the deep convolutional inverse graphics network (DC‐IGN) 8 and U‐Net 9 shown in Figure 1 to efficiently retarget human motion. Our approach handles various types of motions to be retargeted and is sufficiently fast to be used as a real‐time application.…”