Fully Automatic Multi-person Human Motion Capture for VR Applications

Elhayek, Ahmed; Kovalenko, Onorina; Murthy, Pramod; Malik, Jameel; Stricker, Didier

doi:10.1007/978-3-030-01790-3_3

Cited by 18 publications

(12 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using a single camera is obviously more challenging. Works in this area include [23][24][25][26][27][28]. Some works [24,27] predict the 2D joints and then fit the 3D skeleton using different strategies, others [23,25,26] are trained to predict 3D joints directly from the image.…”

Section: Obtaining 3d Poses From Videosmentioning

confidence: 99%

See 1 more Smart Citation

Translating Videos into Synthetic Training Data for Wearable Sensor-Based Activity Recognition Systems Using Residual Deep Convolutional Networks

2021

View full text Add to dashboard Cite

Human activity recognition (HAR) using wearable sensors has benefited much less from recent advances in Deep Learning than fields such as computer vision and natural language processing. This is, to a large extent, due to the lack of large scale (as compared to computer vision) repositories of labeled training data for sensor-based HAR tasks. Thus, for example, ImageNet has images for around 100,000 categories (based on WordNet) with on average 1000 images per category (therefore up to 100,000,000 samples). The Kinetics-700 video activity data set has 650,000 video clips covering 700 different human activities (in total over 1800 h). By contrast, the total length of all sensor-based HAR data sets in the popular UCI machine learning repository is less than 63 h, with around 38 of those consisting of simple mode of locomotion activities like walking, standing or cycling. In our research we aim to facilitate the use of online videos, which exist in ample quantities for most activities and are much easier to label than sensor data, to simulate labeled wearable motion sensor data. In previous work we already demonstrated some preliminary results in this direction, focusing on very simple, activity specific simulation models and a single sensor modality (acceleration norm). In this paper, we show how we can train a regression model on generic motions for both accelerometer and gyro signals and then apply it to videos of the target activities to generate synthetic Inertial Measurement Units (IMU) data (acceleration and gyro norms) that can be used to train and/or improve HAR models. We demonstrate that systems trained on simulated data generated by our regression model can come to within around 10% of the mean F1 score of a system trained on real sensor data. Furthermore, we show that by either including a small amount of real sensor data for model calibration or simply leveraging the fact that (in general) we can easily generate much more simulated data from video than we can collect its real version, the advantage of the latter can eventually be equalized.

show abstract

Section: Obtaining 3d Poses From Videosmentioning

confidence: 99%

“…Works in this area include [23][24][25][26][27][28]. Some works [24,27] predict the 2D joints and then fit the 3D skeleton using different strategies, others [23,25,26] are trained to predict 3D joints directly from the image. Even more interestingly, some methods predict both 2D and 3D keypoints [28].…”

Section: Obtaining 3d Poses From Videosmentioning

confidence: 99%

Translating Videos into Synthetic Training Data for Wearable Sensor-Based Activity Recognition Systems Using Residual Deep Convolutional Networks

2021

View full text Add to dashboard Cite

show abstract

“…Besides, there can be non-rigid objects like human hands whose shape changes over time. Many methods used two-dimensional body parts positions in a monocular image in order to estimate three-dimensional human pose [40,41]. Several studies have been carried out by considering manually labelled two-dimensional body parts positions.…”

Section: Body Shapesmentioning

confidence: 99%

“…The two-dimensional poses of multiple persons were successfully detected thanks to a non-parametric description which helped to learn body parts combinations. Elhayek et al [40] automatically estimated the people number in the scene, and for each image, every three-dimensional skeleton was fitted to equivalent two-dimensional body parts positions calculated thanks to a well-known CNN based twodimensional pose estimation approach. Their method is used to track many individuals in outdoor scenes and in case of lowquality scenes filmed with mobile-phone cameras.…”

Section: Body Shapesmentioning

confidence: 99%

A Novel Spatio-Temporal Violence Classification Framework Based on Material Derivative and LSTM Neural Network

Lejmi¹,

Khalifa²,

Mahjoub³

2020

View full text Add to dashboard Cite

In the current era, the implementation of automated security video surveillance systems is particularly needy in terms of human violence recognition. Nevertheless, the latter encounters various interlinked difficulties which require efficient solutions as well as feasible methods that provide a relevant distinction between normal human actions and abnormal ones. In this paper, we present an overview of these issues and a literature review of the related works and current research on-going efforts on this field and suggests a novel prediction model for violence recognition, based on a preliminary spatio-temporal features extraction using the material derivative which describes the rate of change of a particle while in motion with respect to time. The classification algorithm is then carried out using a deep learning LSTM technique to classify generated features into eight specified violent and non-violent categories and a prediction value for each class of action is calculated. The whole model is trained on a public dataset and its classification capacity is evaluated on a confusion matrix which assembles all the predictions made by the system with their actual labels. The obtained results are promising and show that the proposed model can be potentially useful for detecting human violence.

show abstract

“…Often, a 'penalty term' is used for exceeding joint limits, but these joint limits are worked out from a limited dataset [8,9]. To the best of our knowledge, no recent attempt has been reported to obtain more realistic and statistically valid joint limits from publicly available mocap datasets.…”

Section: Introductionmentioning

confidence: 99%

Learning 3D joint constraints from vision-based motion capture datasets

Murthy

Butt

Hiremath

et al. 2019

IPSJ T Comput Vis Appl

Self Cite

View full text Add to dashboard Cite

Realistic estimation and synthesis of articulated human motion must satisfy anatomical constraints on joint angles. A data-driven approach is used to learn human joint limits from 3D motion capture datasets. We represent joint constraints with a new formulation (s 1 , s 2 , τ) using swing-twist representation in exponential maps form. Our parameterization is applied on Human3.6M dataset to create the lookup-map for each joint. These maps enable us to generate 'synthetic' datasets in entire joint rotation space of a given joint. A set of neural network discriminators is then trained with synthetic datasets to learn valid/invalid joint rotations. The discriminators achieve accuracy of [ 94.4 − 99.4%] for different joints. We validate precision-accuracy trade-off of discriminators and qualitatively evaluate classified poses with an interactive tool. The learned discriminators can be used as 'priors' for human pose estimation and motion synthesis.

show abstract

Fully Automatic Multi-person Human Motion Capture for VR Applications

Cited by 18 publications

References 50 publications

Translating Videos into Synthetic Training Data for Wearable Sensor-Based Activity Recognition Systems Using Residual Deep Convolutional Networks

Translating Videos into Synthetic Training Data for Wearable Sensor-Based Activity Recognition Systems Using Residual Deep Convolutional Networks

A Novel Spatio-Temporal Violence Classification Framework Based on Material Derivative and LSTM Neural Network

Learning 3D joint constraints from vision-based motion capture datasets

Contact Info

Product

Resources

About