Deep Multitask Architecture for Integrated 2D and 3D Human Sensing

Popa, Alin-Ionut; Zanfir, Mihai; Sminchisescu, Cristian

doi:10.1109/cvpr.2017.501

Cited by 151 publications

(97 citation statements)

References 47 publications

(85 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Weakly-supervised learning provides an alternative paradigm for learning robust geometry representation without requiring extensive precise 3D annotation. Most of approaches [45,29,27,41,15] leverage knowledge transformation to learn the robustness by training 3D annotations with abundant 2D annotations in-the-wild. These methods face the difficulties of large domain shift between constrained lab environment for 3D annotations and unconstrained in-the-wild environment for 2D annotations.…”

Section: Introductionmentioning

confidence: 99%

Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation

Chen¹,

Lin

Liu³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

105

View full text Add to dashboard Cite

Recent studies have shown remarkable advances in 3D human pose estimation from monocular images, with the help of large-scale in-door 3D datasets and sophisticated network architectures. However, the generalizability to different environments remains an elusive goal.In this work, we propose a geometry-aware 3D representation for the human pose to address this limitation by using multiple views in a simple auto-encoder model at the training stage and only 2D keypoint information as supervision. A view synthesis framework is proposed to learn the shared 3D representation between viewpoints with synthesizing the human pose from one viewpoint to the other one. Instead of performing a direct transfer in the raw imagelevel, we propose a skeleton-based encoder-decoder mechanism to distil only pose-related representation in the latent space. A learning-based representation consistency constraint is further introduced to facilitate the robustness of latent 3D representation. Since the learnt representation encodes 3D geometry information, mapping it to 3D pose will be much easier than conventional frameworks that use an image or 2D coordinates as the input of 3D pose estimator. We demonstrate our approach on the task of 3D human pose estimation. Comprehensive experiments on three popular benchmarks show that our model can significantly improve the performance of state-of-the-art methods with simply injecting the representation as a robust 3D prior.

show abstract

Section: Introductionmentioning

confidence: 99%

Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation

Chen¹,

Lin

Liu³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

105

View full text Add to dashboard Cite

show abstract

“…Garments are predicted [21] from a single image, but a single model for every new garment needs to be trained, which makes it hard to use in practice. Recent pure bottom-up approaches to human analysis [50,49,58,87,69,71,62] typically predict shape represented as a coarse stick figure or bone skeleton, and can not estimate body shape or clothing.…”

Section: Related Workmentioning

confidence: 99%

Learning to Reconstruct People in Clothing From a Single RGB Camera

Alldieck

Magnor

Bhatnagar

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

322

266

View full text Add to dashboard Cite

Figure 1: We present a deep learning based approach to estimate personalized body shape, including hair and clothing, using a single RGB camera. The shapes shown above have been calculated using only 8 input images, and re-posed using SMPL. AbstractWe present Octopus, a learning-based model to infer the personalized 3D shape of people from a few frames (1-8) of a monocular video in which the person is moving with a reconstruction accuracy of 4 to 5mm, while being orders of magnitude faster than previous methods. From semantic segmentation images, our Octopus model reconstructs a 3D shape, including the parameters of SMPL plus clothing and hair in 10 seconds or less. The model achieves fast and accurate predictions based on two key design choices. First, by predicting shape in a canonical T-pose space, the network learns to encode the images of the person into poseinvariant latent codes, where the information is fused. Second, based on the observation that feed-forward predictions are fast but do not always align with the input images, we predict using both, bottom-up and top-down streams (one per view) allowing information to flow in both directions. Learning relies only on synthetic 3D data. Once learned, Octopus can take a variable number of frames as input, and is able to reconstruct shapes even from a single image with an accuracy of 5mm. Results on 3 different datasets demonstrate the efficacy and accuracy of our approach. Code is available at [2]. * Work partly conducted during an internship at the Real Virtual Humans group of Max Planck for Informatics.

show abstract

“…Recently, deep architectures have been used to learn 3D representations from RGB images [69,57,37,56,38,46] thanks to the availability of high precise 3D data [24], and are now able to surpass depth-sensors [39]. Chen and Ramanan [11] divided the problem of 3D pose estimation into two parts.…”

Section: D Pose Estimationmentioning

confidence: 99%

Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition

Luvizon

Picard

Tabia

2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Human pose estimation and action recognition are related tasks since both problems are strongly dependent on the human body representation and analysis. Nonetheless, most recent methods in the literature handle the two problems separately. In this work, we propose a multi-task framework for jointly estimating 2D or 3D human poses from monocular color images and classifying human actions from video sequences. We show that a single architecture can be used to solve both problems in an efficient way and still achieves state-of-the-art or comparable results at each task while running with a throughput of more than 100 frames per second. The proposed method benefits from high parameters sharing between the two tasks by unifying still images and video clips processing in a single pipeline, allowing the model to be trained with data from different categories simultaneously and in a seamlessly way. Additionally, we provide important insights for end-to-end training the proposed multi-task model by decoupling key prediction parts, which consistently leads to better accuracy on both tasks. The reported results on four datasets (MPII, Human3.6M, Penn Action and NTU RGB+D) demonstrate the effectiveness of our method on the targeted tasks. Our source code and trained weights are publicly available at

show abstract

Deep Multitask Architecture for Integrated 2D and 3D Human Sensing

Cited by 151 publications

References 47 publications

Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation

Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation

Learning to Reconstruct People in Clothing From a Single RGB Camera

Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition

Contact Info

Product

Resources

About