Neural Descent for Visual 3D Human Pose and Shape

Zanfir, Andrei; Bazavan, Eduard Gabriel; Zanfir, Mihai; Freeman, William T.; Sukthankar, Rahul; Sminchisescu, Cristian

doi:10.48550/arxiv.2008.06910

Cited by 5 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Method MPJPE-PA MPJPE HMR [15] 56.8 88.0 GraphCMR [17] 50.1 NR Pose2Mesh [5] 47.0 64.9 I2L-MeshNet [25] 41.7 55.7 SPIN [16] 41.1 NR THUNDR 34.9 48.0 Method MPJPE HMR [15] 89 SPIN [16] 68 HUND [36] 66 THUNDR 53…”

Section: Methodsmentioning

confidence: 99%

“…Moreover, we preserve the spatial structure of high-level image features by avoiding pooling operations, and relying instead on self-attention to enrich our representation [31]. We draw inspiration from vision transformers [8], as we also use a hybrid convolutional-transformer architecture, and from [36], as we explore the idea of iteratively refining estimates by relying on cascaded, input-sensitive processing blocks, with homogeneous parameters.…”

Section: Thundrmentioning

confidence: 99%

“…M 0 are computed based on the default GHUM parameters, (θ 0 , β 0 , r 0 , t 0 ), and camera intrinsics. That is, we find the optimal translation t * 0 such that the corresponding posed mesh projects in the center of the image [36]. Finally, M 0 is computed as…”

Section: Thundrmentioning

confidence: 99%

“…Similarly to [36], we use a soft differentiable rasterizer [21] to compute a body part alignment loss with respect to either ground-truth or predicted body part maps B ∈ R W ×H×15 , with 15 different body part labels…”

Section: Thundrmentioning

confidence: 99%

“…Extensive qualitative testing in the wild supports the overall feasibility, and the quality of 3d reconstructions produced by THUNDR, under both supervised and self-supervised regimes. Related Work: There is considerable prior work in 3d human sensing which we only briefly mention here without aiming at a full literature review [28,4,22,33,26,37,27,15,16,6,5,36]. Methods sometimes referred as 'model-based' [37,16,10,25,14,3,34,39,2,38,9] rely on statistical human body models like SMPL or GHUM, whereas others sometimes referred to as 'model-free' [30,29,13,38] rely on predicting a set of markers or mesh positions, without forms of statistical surface or kinematic regularization based on human anthropometry.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers

Zanfir¹,

Zanfir²,

Bazavan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Figure 1: Automatic 3d pose and shape reconstruction results with THUNDR. (Left) Input image. (Middle) Reconstructed 3d meshes projected on the camera plane and overlayed on the image. (Right) Different viewpoint showing our intermediate predicted marker representation (in green) and the reconstructed surface geometry. THUNDR provides automatic 3D scene placement of the reconstructed humans under a perspective camera model.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Thundrmentioning

confidence: 99%

Section: Thundrmentioning

confidence: 99%

Section: Thundrmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers

Zanfir¹,

Zanfir²,

Bazavan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References

Pang

Chen

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Capturing challenging human motions is critical for numerous applications, but it suffers from complex motion patterns and severe self-occlusion under the monocular setting. In this paper, we propose ChallenCap -a template-based approach to capture challenging 3D human motions using a single RGB camera in a novel learning-and-optimization framework, with the aid of multi-modal references. We propose a hybrid motion inference stage with a generation network, which utilizes a temporal encoder-decoder to extract the motion details from the pair-wise sparse-view reference, as well as a motion discriminator to utilize the unpaired marker-based references to extract specific challenging motion characteristics in a data-driven manner. We further adopt a robust motion optimization stage to increase the tracking accuracy, by jointly utilizing the learned motion details from the supervised multi-modal references as well as the reliable motion hints from the input image reference. Extensive experiments on our new challenging motion dataset demonstrate the effectiveness and robustness of our approach to capture challenging human motions.

show abstract

PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop

Zhang

Tian²,

Zhou

et al. 2021

Preprint

View full text Add to dashboard Cite

Regression-based methods have recently shown promising results in reconstructing human meshes from monocular images. By directly mapping from raw pixels to model parameters, these methods can produce parametric models in a feed-forward manner via neural networks. However, minor deviation in parameters may lead to noticeable misalignment between the estimated meshes and image evidences. To address this issue, we propose a Pyramidal Mesh Alignment Feedback (PyMAF) loop to leverage a feature pyramid and rectify the predicted parameters explicitly based on the mesh-image alignment status in our deep regressor. In PyMAF, given the currently predicted parameters, mesh-aligned evidences will be extracted from finer-resolution features accordingly and fed back for parameter rectification. To reduce noise and enhance the reliability of these evidences, an auxiliary pixel-wise supervision is imposed on the feature encoder, which provides mesh-image correspondence guidance for our network to preserve the most related information in spatial features. The efficacy of our approach is validated on several benchmarks, including Human3.6M, 3DPW, LSP, and COCO, where experimental results show that our approach consistently improves the mesh-image alignment of the reconstruction. Our code is publicly available at https://hongwenzhang.github.io/pymaf.

show abstract

Neural Descent for Visual 3D Human Pose and Shape

Cited by 5 publications

References 0 publications

THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers

THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers

ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References

PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop

Contact Info

Product

Resources

About