Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video

Zhou, Xiaowei; Zhu, Menglong; Leonardos, Spyridon; Derpanis, Konstantinos G.; Daniilidis, Kostas

doi:10.1109/cvpr.2016.537

Cited by 377 publications

(408 citation statements)

References 57 publications

Supporting

Mentioning

403

Contrasting

Order By: Relevance

“…Recent works have made significant advances in the frontier of skeleton-based 3D human pose estimation from single images, with many approaches achieving impressive results [21,23,29,33,35,45]. Although this line of work has boosted the interest for 3D human pose estimation, here we will focus our review on model-based pose estimation.…”

Section: Related Workmentioning

confidence: 99%

Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop

Kolotouros

Pavlakos

Black

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

809

1,031

View full text Add to dashboard Cite

Model-based human pose estimation is currently approached through two different paradigms. Optimizationbased methods fit a parametric body model to 2D observations in an iterative manner, leading to accurate imagemodel alignments, but are often slow and sensitive to the initialization. In contrast, regression-based methods, that use a deep network to directly estimate the model parameters from pixels, tend to provide reasonable, but not pixel accurate, results while requiring huge amounts of supervision. In this work, instead of investigating which approach is better, our key insight is that the two paradigms can form a strong collaboration. A reasonable, directly regressed estimate from the network can initialize the iterative optimization making the fitting faster and more accurate. Similarly, a pixel accurate fit from iterative optimization can act as strong supervision for the network. This is the core of our proposed approach SPIN (SMPL oPtimization IN the loop). The deep network initializes an iterative optimization routine that fits the body model to 2D joints within the training loop, and the fitted estimate is subsequently used to supervise the network. Our approach is self-improving by nature, since better network estimates can lead the optimization to better solutions, while more accurate optimization fits provide better supervision for the network. We demonstrate the effectiveness of our approach in different settings, where 3D ground truth is scarce, or not available, and we consistently outperform the state-of-the-art model-based pose estimation approaches by significant margins. The project website with videos, results, and code can be found at https://seas.upenn.edu/˜nkolot/projects/spin.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop

Kolotouros

Pavlakos

Black

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

809

1,031

View full text Add to dashboard Cite

show abstract

“…Viewpoint parametersR view. Besides the fully supervised methods [25,26], several works have explored multi-view supervision [20,29,31], ordinal depth supervision [28], unpaired 2D-3D data [30,36,41,15] or videos [17] to alleviate the need for full 2D-3D annotations. While these auxiliary sources of supervision allow for compelling 3D predictions, in this work we use only inexpensive 2D keypoint labels.…”

Section: Factorization Networkmentioning

confidence: 99%

C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

Novotný

Ravi

Graham

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

110

177

View full text Add to dashboard Cite

Factorization network Te s t Train ϕ 3D shape and viewpoint (α, θ ) (Y, v) 2D keypoints and visibility Dense keypoints Non-rigid objects Rigid objects Monocular reconstruction of:ϕ Figure 1: Our method learns a 3D model of a deformable object category from 2D keypoints in unconstrained images. It comprises a deep network that learns to factorize shape and viewpoint and, at test time, performs monocular reconstruction. AbstractWe propose C3DPO, a method for extracting 3D models of deformable objects from 2D keypoint annotations in unconstrained images. We do so by learning a deep network that reconstructs a 3D object from a single view at a time, accounting for partial occlusions, and explicitly factoring the effects of viewpoint changes and object deformations. In order to achieve this factorization, we introduce a novel regularization technique. We first show that the factorization is successful if, and only if, there exists a certain canonicalization function of the reconstructed shapes. Then, we learn the canonicalization function together with the reconstruction one, which constrains the result to be consistent. We demonstrate stateof-the-art reconstruction results for methods that do not use ground-truth 3D supervision for a number of benchmarks, including Up3D and PASCAL3D+. Source code has been made available at https

show abstract

“…Learning-based discriminative methods, in particular deep learning methods Lifshitz et al 2016;Newell et al 2016;Tompson et al 2014], represent the current state of the art in 2D pose estimation, with some of these methods demonstrating real-time performance [Cao et al 2016;Wei et al 2016]. Monocular RGB estimation of the 3D skeletal pose is a much harder challenge tackled by relatively fewer methods [Bogo et al 2016;Tekin et al 2016b,c;Zhou et al , 2015b. Unfortunately, these methods are typically offline, and they often reconstruct 3D joint positions individually per image, which are temporally unstable, and do not enforce constant bone lengths.…”

Section: Introductionmentioning

confidence: 99%

VNect

et al. 2017

View full text Add to dashboard Cite

Fig. 1. We recover the full global 3D skeleton pose in real-time from a single RGB camera, even wireless capture is possible by streaming from a smartphone (left). It enables applications such as controlling a game character, embodied VR, sport motion analysis and reconstruction of community video (right). Community videos (CC BY) courtesy of Real Madrid C.F. [2016] and RUSFENCING-TV [2017].We present the first real-time method to capture the full global 3D skeletal pose of a human in a stable, temporally consistent manner using a single RGB camera. Our method combines a new convolutional neural network (CNN) based pose regressor with kinematic skeleton fitting. Our novel fullyconvolutional pose formulation regresses 2D and 3D joint positions jointly in real time and does not require tightly cropped input frames. A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton. This makes our approach the first monocular RGB method usable in real-time applications such as 3D character control-thus far, the only monocular methods for such applications employed specialized RGB-D cameras. Our method's accuracy is quantitatively on par with the best offline 3D monocular RGB pose estimation methods. Our results are qualitatively comparable to, and sometimes better than, results from monocular RGB-D approaches, such as the Kinect. However, we show that our approach is more broadly applicable than RGB-D solutions, i.e., it works for outdoor scenes, community videos, and low quality commodity RGB cameras.

show abstract

Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video

Cited by 377 publications

References 57 publications

Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop

Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop

C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

VNect

Contact Info

Product

Resources

About