3D Human Pose Estimation with Spatial and Temporal Transformers

Zheng, Ce; Zhu, Sijie; Mendieta, Matías; Yang, Taojiannan; Chen, Chen; Ding, Zhengming

doi:10.1109/iccv48922.2021.01145

Cited by 286 publications

(155 citation statements)

References 23 publications

Supporting

Mentioning

151

Contrasting

Order By: Relevance

“…The majority of lifting based approaches work on rootrelative 3D human pose estimation [3,9,12,27,28,34,36,39,51,53,55]. [28] is the pioneer work that introduces the lifting design.…”

Section: Lifting Based 3d Human Pose Estimationmentioning

confidence: 99%

“…[28] is the pioneer work that introduces the lifting design. [3,34,36,53] exploit temporal information to improve the 3D pose estimation accuracy, especially for the occluded cases. Pose ambiguities can be partially resolved by exploiting the temporal context.…”

Section: Lifting Based 3d Human Pose Estimationmentioning

confidence: 99%

“…Human3.6M (H36M) [17] is a large-scale 3D human pose estimation dataset, which contains 3.6 million video frames recorded with four synchronized cameras. Following previous works [34,53], five subjects (S1, S5, S6, S7, S8) and two subjects (S9, S11) with 17-key-point definition are used as training and testing data respectively for SOTA comparison. Humaneva-I [37] is a much smaller dataset compared with H36M , which is captured in a controlled indoor environment with three cameras.…”

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

See 2 more Smart Citations

Ray3D: ray-based 3D human pose estimation for monocular absolute 3D localization

Yu¹,

Weng²,

Choi³

2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose a novel monocular ray-based 3D (Ray3D) absolute human pose estimation with calibrated camera. Accurate and generalizable absolute 3D human pose estimation from monocular 2D pose input is an ill-posed problem. To address this challenge, we convert the input from pixel space to 3D normalized rays. This conversion makes our approach robust to camera intrinsic parameter changes. To deal with the in-the-wild camera extrinsic parameter variations, Ray3D explicitly takes the camera extrinsic parameters as an input and jointly models the distribution between the 3D pose rays and camera extrinsic parameters. This novel network design is the key to the outstanding generalizability of Ray3D approach. To have a comprehensive understanding of how the camera intrinsic and extrinsic parameter variations affect the accuracy of absolute 3D key-point localization, we conduct in-depth systematic experiments on three single person 3D benchmarks as well as one synthetic benchmark. These experiments demonstrate that our method significantly outperforms existing state-of-the-art models. Our code and the synthetic dataset are available at https://github.com/YxZhxn

show abstract

“…The majority of lifting based approaches work on rootrelative 3D human pose estimation [3,9,12,27,28,34,36,39,51,53,55]. [28] is the pioneer work that introduces the lifting design.…”

Section: Lifting Based 3d Human Pose Estimationmentioning

confidence: 99%

Section: Lifting Based 3d Human Pose Estimationmentioning

confidence: 99%

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

See 1 more Smart Citation

Ray3D: ray-based 3D human pose estimation for monocular absolute 3D localization

Yu¹,

Weng²,

Choi³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Nevertheless, the information in a single image is far from sufficient considering the occlusion and depth ambiguity. For compensation, some works [13], [14], [15], [16], [17] utilize temporal information from video sequences. Sequential variation in the video is conducive to reveal the structure of the human body.…”

Section: Introductionmentioning

confidence: 99%

“…• Hui Shuai, Lele Wu, and Qingshan Liu are with Nanjing University of Information Science and Technology, 210044 E-mail: huishuai13@nuist.com, llwu@nuist.edu.cn, qsliu@nuist.edu.cn [48]  FCN [25]  Cascaded [19]  GOR [34]  SRNeT [44]  MDN [17]  Skeletal-GNN [45]  ViewPose3D [30]  OAN [4]  SRNeT [44]  Motion-guided [42]  PoseFormer [49]  Skeletal-GNN [45]  Epipolar Transformer [7]  CrossView [32]  Learnable [10]  AdaFuse [47]  FLEX [6]  DeepFuse [8]  FLEX [6] Fig. 1.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation

Shuai,

Wu,

Liu

2021

Preprint

View full text Add to dashboard Cite

In practical application, 3D Human Pose Estimation (HPE) is facing with several variable elements, involving the number of views, the length of the video sequence, and whether using camera calibration. To this end, we propose a unified framework named Multi-view and Temporal Fusing Transformer (MTF-Transformer) to adaptively handle varying view numbers and video length without calibration. MTF-Transformer consists of Feature Extractor, Multi-view Fusing Transformer (MFT), and Temporal Fusing Transformer (TFT). Feature Extractor estimates the 2D pose from each image and encodes the predicted coordinates and confidence into feature embedding for further 3D pose inference. It discards the image features and focuses on lifting the 2D pose into the 3D pose, making the subsequent modules computationally lightweight enough to handle videos. MFT fuses the features of a varying number of views with a relative-attention block. It adaptively measures the implicit relationship between each pair of views and reconstructs the features. TFT aggregates the features of the whole sequence and predicts 3D pose via a transformer, which is adaptive to the length of the video and takes full advantage of the temporal information. With these modules, MTF-Transformer handles different application scenes, varying from a monocular-single-image to multi-view-video, and the camera calibration is avoidable. We demonstrate quantitative and qualitative results on the Human3.6M, TotalCapture, and KTH Multiview Football II. Compared with state-of-the-art methods with camera parameters, experiments show that MTF-Transformer not only obtains comparable results but also generalizes well to dynamic capture with an arbitrary number of unseen views. Code is available in https://github.com/lelexx/MTF-Transformer.

show abstract

Parallel‐branch network for 3D human pose and shape estimation in video

Wang

2022

Computer Animation & Virtual

View full text Add to dashboard Cite

Human pose and shape estimation have developed rapidly, where a skinned multi-person linear (SMPL) approach performs excellent recently. However, the prior template of the human body in the SMPL model is fixed, thus a deviation may be resulted in the reconstructed body shape if a human body acts sharp movements such as sporting or dancing. To address this problem, we propose a parallel-branch network including a designed spatial-temporal (ST) branch and a SMPL branch. The ST branch essentially performs the 2D-to-3D lifting for more accurate joint prediction, by the designed spatial transformer and temporal transformer. The 3D joints from the ST branch are used to supervise the 3D joints from the SMPL branch and further correct the deviation of the SMPL model. Experiments on some popular benchmarks like 3DPW and MPI-INF-3DHP show that our method has better performance than other methods with video input. Our code is available at https://automation.seu.edu.cn/ wcx/list.htm

show abstract

3D Human Pose Estimation with Spatial and Temporal Transformers

Cited by 286 publications

References 23 publications

Ray3D: ray-based 3D human pose estimation for monocular absolute 3D localization

Ray3D: ray-based 3D human pose estimation for monocular absolute 3D localization

Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation

Parallel‐branch network for 3D human pose and shape estimation in video

Contact Info

Product

Resources

About