Parallel‐branch network for 3D human pose and shape estimation in video

Abstract

Human pose and shape estimation have developed rapidly, where a skinned multi-person linear (SMPL) approach performs excellent recently. However, the prior template of the human body in the SMPL model is fixed, thus a deviation may be resulted in the reconstructed body shape if a human body acts sharp movements such as sporting or dancing. To address this problem, we propose a parallel-branch network including a designed spatial-temporal (ST) branch and a SMPL branch. The ST branch essentially performs the 2D-to-3D lifting for more accurate joint prediction, by the designed spatial transformer and temporal transformer. The 3D joints from the ST branch are used to supervise the 3D joints from the SMPL branch and further correct the deviation of the SMPL model. Experiments on some popular benchmarks like 3DPW and MPI-INF-3DHP show that our method has better performance than other methods with video input. Our code is available at https://automation.seu.edu.cn/ wcx/list.htm

Physical based motion reconstruction from videos using musculoskeletal model

“…TCMR 18 significantly improves the time consistency between pose estimation results, but it suffers a loss of spatial accuracy. Sun et al proposed a method called BTMR 3 to strike a balance between temporal continuity and spatial accuracy, and as a result, we adopt BTMR as the pose estimation method in our paper.…”

“…The learning process consists of two phases: pose estimation and motion imitation. During the pose estimation phase, a pre‐trained 3D pose estimator 3 is used to extract the pose of the character in the video as a reference motion. This reference motion is generated by solving inverse kinematics based on the 3D human joint positions output by the pose estimator.…”

“…We use a pre‐trained BTMR model 3 for pose estimation, which consists of a static feature extractor, a bidirectional temporal feature network including bidirectional temporal encoder and bidirectional temporal integration network, an SMPL 30 parameter regressor and a motion discriminator. Given a human motion video sequence as the input, the static features of each frame are first extracted using a pre‐trained convolutional neural network ResNet 31 .…”

“…The motions generated by joint modeling methods are not as realistic as those generated by musculoskeletal modeling approaches, although joint modeling methods reduce model complexity and provide better generalization and stability. Therefore, we propose a novel approach to reconstruct human motion from videos, which combines video pose estimation methods 3 with muscle‐driven character control methods based on the requirement for a wider and more accessible motion data source and more realistic motion effects.…”