Synthesizing Training Images for Boosting Human 3D Pose Estimation

Chen, Wenzheng; Wang, Huan; Li, Yangyan; Su, Hao; Wang, Zhenhua; Tu, Changhe; Lischinski, Dani; Cohen–Or, Daniel; Chen, Baoquan

doi:10.1109/3dv.2016.58

Cited by 255 publications

(219 citation statements)

References 52 publications

Supporting

Mentioning

217

Contrasting

Order By: Relevance

“…Even the most recent annotated real 3D pose data sets, or combined real/synthetic data sets [Chen et al 2016;Ionescu et al 2014b;Mehta et al 2016] are a subset of real world human pose, shape, appearance and background distributions. Recent top performing methods explicitly address this data sparsity by training similarly deep networks, but with architectural changes enabling improved intermediate training supervision [Mehta et al 2016].…”

Section: Discussionmentioning

confidence: 99%

VNect

et al. 2017

View full text Add to dashboard Cite

Fig. 1. We recover the full global 3D skeleton pose in real-time from a single RGB camera, even wireless capture is possible by streaming from a smartphone (left). It enables applications such as controlling a game character, embodied VR, sport motion analysis and reconstruction of community video (right). Community videos (CC BY) courtesy of Real Madrid C.F. [2016] and RUSFENCING-TV [2017].We present the first real-time method to capture the full global 3D skeletal pose of a human in a stable, temporally consistent manner using a single RGB camera. Our method combines a new convolutional neural network (CNN) based pose regressor with kinematic skeleton fitting. Our novel fullyconvolutional pose formulation regresses 2D and 3D joint positions jointly in real time and does not require tightly cropped input frames. A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton. This makes our approach the first monocular RGB method usable in real-time applications such as 3D character control-thus far, the only monocular methods for such applications employed specialized RGB-D cameras. Our method's accuracy is quantitatively on par with the best offline 3D monocular RGB pose estimation methods. Our results are qualitatively comparable to, and sometimes better than, results from monocular RGB-D approaches, such as the Kinect. However, we show that our approach is more broadly applicable than RGB-D solutions, i.e., it works for outdoor scenes, community videos, and low quality commodity RGB cameras.

show abstract

Section: Discussionmentioning

confidence: 99%

VNect

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Recent works have shown the success of deep network architectures for the problem of retrieving 3D features such as kinematic joints [4,33] or surface characterizations [43] from single images, with extremely encouraging results. Such successes, sometimes achieved with simple, standard network architectures [30], have naturally motivated Figure 1.…”

Section: Introductionmentioning

confidence: 99%

Moulding Humans: Non-Parametric 3D Human Shape Estimation From Single Images

Gabeur

Franco

Martín

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

166

111

View full text Add to dashboard Cite

In this paper, we tackle the problem of 3D human shape estimation from single RGB images. While the recent progress in convolutional neural networks has allowed impressive results for 3D human pose estimation, estimating the full 3D shape of a person is still an open issue. Model-based approaches can output precise meshes of naked under-cloth human bodies but fail to estimate details and un-modelled elements such as hair or clothing. On the other hand, non-parametric volumetric approaches can potentially estimate complete shapes but, in practice, they are limited by the resolution of the output grid and cannot produce detailed estimates. In this work, we propose a non-parametric approach that employs a double depth map to represent the 3D shape of a person: a visible depth map and a "hidden" depth map are estimated and combined, to reconstruct the human 3D shape as done with a "mould". This representation through 2D depth maps allows a higher resolution output with a much lower dimension than voxel-based volumetric representations. Additionally, our fully derivable depth-based model allows us to efficiently incorporate a discriminator in an adversarial fashion to improve the accuracy and "humanness" of the 3D output. We train and quantitatively validate our approach on SURREAL and on 3D-HUMANS, a new photorealistic dataset made of semi-synthetic in-house images annotated with 3D ground truth surfaces.

show abstract

“…Therefore, researchers have also investigated the use of computer graphics to synthesize large datasets for applications requiring a different set of annotations, e.g. human 3D pose estimation [12], [13] or 3D character manipulation [14]. The key element with synthetic images is that annotations come at no cost since these are generated during rendering.…”

Section: A Motivationsmentioning

confidence: 99%

Efficient Convolutional Neural Networks for Depth-Based Multi-Person Pose Estimation

Martínez-González

Villamizar

Canévet

et al. 2020

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Achieving robust multi-person 2D body landmark localization and pose estimation is essential for human behavior and interaction understanding as encountered for instance in HRI settings. Accurate methods have been proposed recently, but they usually rely on rather deep Convolutional Neural Network (CNN) architecture, thus requiring large computational and training resources. In this paper, we investigate different architectures and methodologies to address these issues and achieve fast and accurate multi-person 2D pose estimation. To foster speed, we propose to work with depth images, whose structure contains sufficient information about body landmarks while being simpler than textured color images and thus potentially requiring less complex CNNs for processing. In this context, we make the following contributions. i) we study several CNN architecture designs combining pose machines relying on the cascade of detectors concept with lightweight and efficient CNN structures; ii) to address the need for large training datasets with high variability, we rely on semi-synthetic data combining multi-person synthetic depth data with real sensor backgrounds; iii) we explore domain adaptation techniques to address the performance gap introduced by testing on real depth images; iv) to increase the accuracy of our fast lightweight CNN models, we investigate knowledge distillation at several architecture levels which effectively enhance performance. Experiments and results on synthetic and real data highlight the impact of our design choices, providing insights into methods addressing standard issues normally faced in practical applications, and resulting in architectures effectively matching our goal in both performance and speed.Index Terms-Human Pose Estimation, Convolutional Neural Networks, Machine Learning.CNN-based human pose estimation methods traditionally use a deep architecture pretrained on a large scale image recognition dataset. This design choice might unnecessary bring high computational burden. In this paper, inspired by efficient network structures such as those encountered in ResNets [7], MobileNets [8] and SqueezeNets [9], we introduce novel lightweight network architectures that match our real-time

show abstract

Synthesizing Training Images for Boosting Human 3D Pose Estimation

Cited by 255 publications

References 52 publications

VNect

VNect

Moulding Humans: Non-Parametric 3D Human Shape Estimation From Single Images

Efficient Convolutional Neural Networks for Depth-Based Multi-Person Pose Estimation

Contact Info

Product

Resources

About