3D Human Pose Estimation with Spatial and Temporal Transformers

Zheng, Ce; Zhu, Sijie; Mendieta, Matías; Yang, Taojiannan; Chen, Chen; Ding, Zhengming

doi:10.48550/arxiv.2103.10455

Cited by 10 publications

(13 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, transformers are especially good at modelling long-range dependencies between elements of a sequence. Since then, there have been several attempts to adapt transformers towards vision tasks including object detection [2,56], image classification [8,41,52,46,14], segmentation [44], multiple object tracking [37,29], human pose estimation [50,55], point cloud processing [12,54], video processing [10,31,38], image super-resolution [30,49,3], image synthesis [9], etc. An extensive review is out of the scope of this paper.…”

Section: Transformers and Vision Transformersmentioning

confidence: 99%

LocalViT: Bringing Locality to Vision Transformers

Li,

Zhang,

Cao

et al. 2021

Preprint

124

146

View full text Add to dashboard Cite

We study how to introduce locality mechanisms into vision transformers. The transformer network originates from machine translation and is particularly good at modelling long-range dependencies within a long sequence. Although the global interaction between the token embeddings could be well modelled by the self-attention mechanism of transformers, what is lacking a locality mechanism for information exchange within a local region. Yet, locality is essential for images since it pertains to structures like lines, edges, shapes, and even objects.We add locality to vision transformers by introducing depth-wise convolution into the feed-forward network. This seemingly simple solution is inspired by the comparison between feed-forward networks and inverted residual blocks. The importance of locality mechanisms is validated in two ways: 1) A wide range of design choices (activation function, layer placement, expansion ratio) are available for incorporating locality mechanisms and all proper choices can lead to a performance gain over the baseline, and 2) The same locality mechanism is successfully applied to 4 vision transformers, which shows the generalization of the locality concept. In particular, for ImageNet2012 classification, the locality-enhanced transformers outperform the baselines DeiT-T [41] and PVT-T [46] by 2.6% and 3.1% with a negligible increase in the number of parameters and computational effort. Code is available at https://github.com/ofsoundof/LocalViT.

show abstract

Section: Transformers and Vision Transformersmentioning

confidence: 99%

LocalViT: Bringing Locality to Vision Transformers

Li,

Zhang,

Cao

et al. 2021

Preprint

124

146

View full text Add to dashboard Cite

show abstract

“…Human Pose Estimation. Recent several works [28,16,19,40,44,29] introduce Transformer for human pose es-…”

Section: Vision Transformermentioning

confidence: 99%

TokenPose: Learning Keypoint Tokens for Human Pose Estimation

Li¹,

Zhang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Human pose estimation deeply relies on visual clues and anatomical constraints between parts to locate keypoints. Most existing CNN-based methods do well in visual representation, however, lacking in the ability to explicitly learn the constraint relationships between keypoints. In this paper, we propose a novel approach based on Token representation for human Pose estimation (TokenPose). In detail, each keypoint is explicitly embedded as a token to simultaneously learn constraint relationships and appearance cues from images. Extensive experiments show that the small and large TokenPose models are on par with state-of-the-art CNN-based counterparts while being more lightweight. Specifically, our TokenPose-S and TokenPose-L achieve 72.5 AP and 75.8 AP on COCO validation dataset respectively, with significant reduction in parameters (↓ 80.6% ; ↓ 56.8%) and GFLOPs (↓75.3%; ↓ 24.7%).

show abstract

“…Vision Transformer Recently, several studies demonstrated that the transformer architectures [46] plays a significant role in a wide range of computer vision tasks, such as image classification [4,11,45], object detection [2,64], and semantic segmentation [48,55,63]. Recently, some studies also explored applying the transformer on human pose estimation tasks [28,29,31,56,62]. More specifically, for 2D pose estimation, TransPose [56] aims to explain the spatial dependencies of the predicted keypoints with transformers, PRTR [28] and TF-Pose [31] attempt to directly regress the joint coordinates by transformer decoders.…”

Section: Transformermentioning

confidence: 99%

“…More specifically, for 2D pose estimation, TransPose [56] aims to explain the spatial dependencies of the predicted keypoints with transformers, PRTR [28] and TF-Pose [31] attempt to directly regress the joint coordinates by transformer decoders. While for the 3D pose estimation, METRO [29] firstly applies transformer to reconstruct 3D human pose and mesh from a single image, and PoseFormer [62] builds a spatial-temporal transformers with the input of 2D joint sequences for 3D pose estimation in videos. However, previous works have hardly exploited the transformer architectures on the multi-view 3D pose estimation setting, which is however an important task in the pose estimation area.…”

Section: Transformermentioning

confidence: 99%

TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation

Ma¹,

Chen²,

Kong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Estimating the 2D human poses in each view is typically the first step in calibrated multi-view 3D pose estimation. But the performance of 2D pose detectors suffers from challenging situations such as occlusions and oblique viewing angles. To address these challenges, previous works derive point-to-point correspondences between different views from epipolar geometry and utilize the correspondences to merge prediction heatmaps or feature representations. Instead of post-prediction merge/calibration, here we introduce a transformer framework for multi-view 3D pose estimation, aiming at directly improving individual 2D predictors by integrating information from different views. Inspired by previous multi-modal transformers, we design a unified transformer architecture, named TransFusion, to fuse cues from both current views and neighboring views. Moreover, we propose the concept of epipolar field to encode 3D positional information into the transformer model. The 3D position encoding guided by epipolar field provides an efficient way of encoding correspondences between pixels of different views. Experiments on Human 3.6M and Ski-Pose show that our method is more efficient and has consistent improvements compared to other fusion methods. Specifically, we achieve 25.8 mm

show abstract

3D Human Pose Estimation with Spatial and Temporal Transformers

Cited by 10 publications

References 34 publications

LocalViT: Bringing Locality to Vision Transformers

LocalViT: Bringing Locality to Vision Transformers

TokenPose: Learning Keypoint Tokens for Human Pose Estimation

TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation

Contact Info

Product

Resources

About