Deep Autoencoder for Combined Human Pose Estimation and Body Model Upscaling

Trumble, Matthew; Gilbert, Andrew; Hilton, Adrian; Collomosse, John

doi:10.1007/978-3-030-01249-6_48

Cited by 54 publications

(39 citation statements)

References 48 publications

(61 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3.5, Tri CPM LSTM. As one can see from Table 2, our proposed approach outperforms all compared methods at time of publication [the newer works of Martinez et al (2017) and Trumble et al (2018)] indicate the speed of improvement in field of 3D pose estimation) despite excluding the fusion with the kinematic based IMU, with the mean error reduced by 15% compared with (Tome et al 2017), the Tri CPM LSTM approach and our previous method . While compared to the state of the art results by Mude Lin Liang Lin and Cheng (2017), many activities have a similar error around 5 or 6cm.…”

Section: Human 36mmentioning

confidence: 86%

“…Given these restrictions, we propose a new dataset to address these short comings, TotalCapture. 1 It contains a large amount of MVV, and synchronised IMU Table 2 A comparison of our approach to other works on the Human 3.6M dataset, multiview indicates whether the approach uses multiple camera views [the works of Martinez et al (2017) and Trumble et al (2018) and Vicon labelling for ground truth. It was captured indoors in a volume measuring roughly 8 × 4 m with 8 calibrated HD video cameras at 60 Hz.…”

Section: Total Capturementioning

confidence: 99%

See 1 more Smart Citation

Fusing Visual and Inertial Sensors with Semantics for 3D Human Pose Estimation

et al. 2018

Self Cite

View full text Add to dashboard Cite

We propose an approach to accurately estimate 3D human pose by fusing multi-viewpoint video (MVV) with inertial measurement unit (IMU) sensor data, without optical markers, a complex hardware setup or a full body model. Uniquely we use a multi-channel 3D convolutional neural network to learn a pose embedding from visual occupancy and semantic 2D pose estimates from the MVV in a discretised volumetric probabilistic visual hull. The learnt pose stream is concurrently processed with a forward kinematic solve of the IMU data and a temporal model (LSTM) exploits the rich spatial and temporal long range dependencies among the solved joints, the two streams are then fused in a final fully connected layer. The two complementary data sources allow for ambiguities to be resolved within each sensor modality, yielding improved accuracy over prior methods. Extensive evaluation is performed with state of the art performance reported on the popular Human 3.6M dataset (Ionescu et al. in Intell IEEE Trans Pattern Anal Mach 36(7):1325-1339, 2014), the newly released TotalCapture dataset and a challenging set of outdoor videos TotalCaptureOutdoor. We release the new hybrid MVV dataset (TotalCapture) comprising of multi-viewpoint video, IMU and accurate 3D skeletal joint ground truth derived from a commercial motion capture system.

show abstract

Section: Human 36mmentioning

confidence: 86%

Section: Total Capturementioning

confidence: 99%

Fusing Visual and Inertial Sensors with Semantics for 3D Human Pose Estimation

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…Most approaches take a two-stage approach: first obtaining a singleview 3D reconstruction and then post-processing the result to be smooth via solving a constrained optimization problem [66,58,46,47,27,38,43]. Recent methods obtain accurate shapes and textures of clothing by pre-capturing the actors and making use of silhouettes [50,60,24,4]. While these approaches obtain far more accurate shape, reliance on the pre-scan and silhouettes restricts these approaches to videos obtained in an interactive and controlled environments.…”

Section: Related Workmentioning

confidence: 99%

Learning 3D Human Dynamics From Video

Kanazawa

Zhang

Felsen

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

469

454

View full text Add to dashboard Cite

From an image of a person in action, we can easily guess the 3D motion of the person in the immediate past and future. This is because we have a mental model of 3D human dynamics that we have acquired from observing visual sequences of humans in motion. We present a framework that can similarly learn a representation of 3D dynamics of humans from video via a simple but effective temporal encoding of image features. At test time, from video, the learned temporal representation give rise to smooth 3D mesh predictions. From a single image, our model can recover the current 3D mesh as well as its 3D past and future motion. Our approach is designed so it can learn from videos with 2D pose annotations in a semi-supervised manner. Though annotated data is always limited, there are millions of videos uploaded daily on the Internet. In this work, we harvest this Internet-scale source of unlabeled data by training our model on unlabeled video with pseudo-ground truth 2D pose obtained from an off-the-shelf 2D pose detector. Our experiments show that adding more videos with pseudo-ground truth 2D pose monotonically improves 3D prediction performance. We evaluate our model, Human Mesh and Motion Recovery (HMMR), on the recent challenging dataset of 3D Poses in the Wild and obtain state-of-the-art performance on the 3D prediction task without any fine-tuning. The project website with video, code, and data can be found at https://akanazawa.github.io/ human_dynamics/.

show abstract

“…Studies have shown that human motion includes a lot of redundant information and can be approximately presented by dimensions lower than the original degree of freedom of human motion [ 6 , 17 , 18 ], which inspired the study of sparse sensor motion capture. Some researchers developed motion-capture technologies combining sparse inertial sensors with video input [ 7 , 19 ] or optical reflective markers [ 20 ] or using only sparse optical markers [ 21 ].…”

Section: Related Workmentioning

confidence: 99%

Training Data Selection and Optimal Sensor Placement for Deep-Learning-Based Sparse Inertial Sensor Human Posture Reconstruction

Zheng

Yan

et al. 2021

Entropy

View full text Add to dashboard Cite

Although commercial motion-capture systems have been widely used in various applications, the complex setup limits their application scenarios for ordinary consumers. To overcome the drawbacks of wearability, human posture reconstruction based on a few wearable sensors have been actively studied in recent years. In this paper, we propose a deep-learning-based sparse inertial sensor human posture reconstruction method. This method uses bidirectional recurrent neural network (Bi-RNN) to build an a priori model from a large motion dataset to build human motion, thereby the low-dimensional motion measurements are mapped to whole-body posture. To improve the motion reconstruction performance for specific application scenarios, two fundamental problems in the model construction are investigated: training data selection and sparse sensor placement. The problem of deep-learning training data selection is to select independent and identically distributed (IID) data for a certain scenario from the accumulated imbalanced motion dataset with sufficient information. We formulate the data selection into an optimization problem to obtain continuous and IID data segments, which comply with a small reference dataset collected from the target scenario. A two-step heuristic algorithm is proposed to solve the data selection problem. On the other hand, the optimal sensor placement problem is studied to exploit most information from partial observation of human movement. A method for evaluating the motion information amount of any group of wearable inertial sensors based on mutual information is proposed, and a greedy searching method is adopted to obtain the approximate optimal sensor placement of a given sensor number, so that the maximum motion information and minimum redundancy is achieved. Finally, the human posture reconstruction performance is evaluated with different training data and sensor placement selection methods, and experimental results show that the proposed method takes advantages in both posture reconstruction accuracy and model training time. In the 6 sensors configuration, the posture reconstruction errors of our model for walking, running, and playing basketball are 7.25, 8.84, and 14.13, respectively.

show abstract

Deep Autoencoder for Combined Human Pose Estimation and Body Model Upscaling

Cited by 54 publications

References 48 publications

Fusing Visual and Inertial Sensors with Semantics for 3D Human Pose Estimation

Fusing Visual and Inertial Sensors with Semantics for 3D Human Pose Estimation

Learning 3D Human Dynamics From Video

Training Data Selection and Optimal Sensor Placement for Deep-Learning-Based Sparse Inertial Sensor Human Posture Reconstruction

Contact Info

Product

Resources

About