Bidirectional temporal feature for 3D human pose and shape estimation from a video

Abstract

3D human pose and shape estimation is the foundation of analyzing human motion. However, estimating accurate and temporally consistent 3D human motion from a video remains a challenge. By now, most of the video-based methods for estimating 3D human pose and shape rely on unidirectional temporal features and lack more comprehensive information. To solve this problem, we propose a novel model "bidirectional temporal feature for human motion recovery" (BTMR), which consists of a human motion generator and a discriminator. The transformer-based generator effectively captures the forward and reverse temporal features to enhance the temporal correlation between frames and reduces the loss of spatial information. The motion discriminator based on Bi-LSTM can distinguish whether the generated pose sequences are consistent with the realistic sequences of the AMASS dataset. In the process of continuous generation and discrimination, the model can learn more realistic and accurate poses. We evaluate our BTMR on 3DPW and MPI-INF-3DHP datasets. Without the training set of 3DPW, BTMR outperforms VIBE by 2.4 mm and 14.9 mm/s 2 in PA-MPJPE and Accel metrics and outperforms TCMR by 1.7 mm in PA-MPJPE metric on 3DPW. The results demonstrate that our BTMR produces better accurate and temporal consistent 3D human motion.

A Modal Fusion Network with Dual Attention Mechanism for 6D Pose Estimation

“…This enables better prediction of the three-dimensional coordinates of occluded pixels. A model proposed by Libo Sun [29] and colleagues utilizes bidirectional temporal features (BTMR) to estimate human motion poses based on image sequences. This model learns more realistic and accurate poses in the continuous generation and discrimination process.…”

Physical based motion reconstruction from videos using musculoskeletal model

“…TCMR 18 significantly improves the time consistency between pose estimation results, but it suffers a loss of spatial accuracy. Sun et al proposed a method called BTMR 3 to strike a balance between temporal continuity and spatial accuracy, and as a result, we adopt BTMR as the pose estimation method in our paper.…”

“…The learning process consists of two phases: pose estimation and motion imitation. During the pose estimation phase, a pre‐trained 3D pose estimator 3 is used to extract the pose of the character in the video as a reference motion. This reference motion is generated by solving inverse kinematics based on the 3D human joint positions output by the pose estimator.…”

“…We use a pre‐trained BTMR model 3 for pose estimation, which consists of a static feature extractor, a bidirectional temporal feature network including bidirectional temporal encoder and bidirectional temporal integration network, an SMPL 30 parameter regressor and a motion discriminator. Given a human motion video sequence as the input, the static features of each frame are first extracted using a pre‐trained convolutional neural network ResNet 31 .…”

“…The motions generated by joint modeling methods are not as realistic as those generated by musculoskeletal modeling approaches, although joint modeling methods reduce model complexity and provide better generalization and stability. Therefore, we propose a novel approach to reconstruct human motion from videos, which combines video pose estimation methods 3 with muscle‐driven character control methods based on the requirement for a wider and more accessible motion data source and more realistic motion effects.…”