Deep inertial poser

Huang, Yinghao; Kaufmann, Manuel; Aksan, Emre; Black, Michael J.; Hilliges, Otmar; Pons-Moll, Gerard

doi:10.1145/3272127.3275108

Cited by 212 publications

(161 citation statements)

References 71 publications

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…To learn from sufficient data and incorporate variations of movement, Johnson et al (2019) synthesized accelerometer data via double-differentiation of marker trajectories from their OMC archive. Huang et al (2018) also synthesized inertial sensor data from motion capture datasets using a 3D model of the human body shape and pose (SMPL) together with a virtual sensor model. Mundt et al (2020a,b) used OMC data from several studies of their lab together with a biomechanical model to create a large simulated dataset, which was used for training feedforward neural networks to estimate joint kinematics and kinetics.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CNN-Based Estimation of Sagittal Plane Walking and Running Biomechanics From Measured and Simulated Inertial Sensor Data

Dorschky

Nitschke

Martindale

et al. 2020

Front. Bioeng. Biotechnol.

View full text Add to dashboard Cite

Machine learning is a promising approach to evaluate human movement based on wearable sensor data. A representative dataset for training data-driven models is crucial to ensure that the model generalizes well to unseen data. However, the acquisition of sufficient data is time-consuming and often infeasible. We present a method to create realistic inertial sensor data with corresponding biomechanical variables by 2D walking and running simulations. We augmented a measured inertial sensor dataset with simulated data for the training of convolutional neural networks to estimate sagittal plane joint angles, joint moments, and ground reaction forces (GRFs) of walking and running. When adding simulated data, the root mean square error (RMSE) of the test set of hip, knee, and ankle joint angles decreased up to 17 %, 27 % and 23 %, the RMSE of knee and ankle joint moments up to 6 % and the RMSE of anterior-posterior and vertical GRF up to 2 and 6 %. Simulation-aided estimation of joint moments and GRFs was limited by inaccuracies of the biomechanical model. Improving the physics-based model and domain adaptation learning may further increase the benefit of simulated data. Future work can exploit biomechanical simulations to connect different data sources in order to create representative datasets of human movement. In conclusion, machine learning can benefit from available domain knowledge on biomechanical simulations to supplement cumbersome data collections.

show abstract

Section: Introductionmentioning

confidence: 99%

“…One drawback of these approaches is that additional datasets containing OMC data or SMPL poses of the movement of interest were required. Notably, Huang et al (2018) reported that combining these datasets was nontrivial. Moreover, each recorded motion trajectory led to only one synthetic sensor trajectory.…”

Section: Introductionmentioning

confidence: 99%

CNN-Based Estimation of Sagittal Plane Walking and Running Biomechanics From Measured and Simulated Inertial Sensor Data

Dorschky

Nitschke

Martindale

et al. 2020

Front. Bioeng. Biotechnol.

View full text Add to dashboard Cite

show abstract

“…The neck and shoulder bones move depending on the chest, while the hip bones move depending on the pelvis. Additionally, various other bones, like the feet, hands, fingers, and head, are merely extensions in the hierarchy by either increasing the number of sensors or by adopting deep learning methods like in the [28,29]. The Motion-sphere could be extended to these bone segments, depending on the application.…”

Section: Discussionmentioning

confidence: 99%

Motion-Sphere: Visual Representation of the Subtle Motion of Human Joints

et al. 2020

View full text Add to dashboard Cite

Understanding and differentiating subtle human motion over time as sequential data is challenging. We propose Motion-sphere, which is a novel trajectory-based visualization technique, to represent human motion on a unit sphere. Motion-sphere adopts a two-fold approach for human motion visualization, namely a three-dimensional (3D) avatar to reconstruct the target motion and an interactive 3D unit sphere, that enables users to perceive subtle human motion as swing trajectories and color-coded miniature 3D models for twist. This also allows for the simultaneous visual comparison of two motions. Therefore, the technique is applicable in a wide range of applications, including rehabilitation, choreography, and physical fitness training. The current work validates the effectiveness of the proposed work with a user study in comparison with existing motion visualization methods. Our study’s findings show that Motion-sphere is informative in terms of quantifying the swing and twist movements. The Motion-sphere is validated in threefold ways: validation of motion reconstruction on the avatar, accuracy of swing, twist, and speed visualization, and the usability and learnability of the Motion-sphere. Multiple range of motions from an online open database are selectively chosen, such that all joint segments are covered. In all fronts, Motion-sphere fares well. Visualization on the 3D unit sphere and the reconstructed 3D avatar make it intuitive to understand the nature of human motion.

show abstract

“…Existing solutions are mostly based on the installation of new cameras, either based on retro-reflective markers [4] or computer vision techniques (e.g., Microsoft Kinect) [5,6] to estimate human pose. Other common methods rely on body-worn suits with a set of IMU sensors to reconstruct body pose [3]. While effective, these approaches can increase the cost of the XR system rapidly.…”

Section: Introductionmentioning

confidence: 99%

ShadowDancXR: Body Gesture Digitization for Low-cost Extended Reality (XR) Headsets

Jiang

Wang

et al. 2020

Companion Proceedings of the 2020 Conference on Interactive Surfaces and Spaces

View full text Add to dashboard Cite

Figure 1: (a) ShadowDancXR uses the front-facing camera of the smartphone in low-cost XR headsets (e.g., Google Cardboard) to reconstruct users' body gestures through projected shadows. We designed two interactive experiences based on Shadow-DancXR: (b) allows users to see themselves as robot Kyle who will mimic their body postures through VR Mirror, and (c) enables users to "fly" around a virtual world as a bird with multiple arm postures.

show abstract

Deep inertial poser

Abstract: constitutes the largest IMU dataset publicly available. We quantitatively evaluate our approach on multiple datasets and show results from a real-time implementation. DIP-IMU and the code are available for research purposes. 1

Cited by 212 publications

References 71 publications

CNN-Based Estimation of Sagittal Plane Walking and Running Biomechanics From Measured and Simulated Inertial Sensor Data

CNN-Based Estimation of Sagittal Plane Walking and Running Biomechanics From Measured and Simulated Inertial Sensor Data

Motion-Sphere: Visual Representation of the Subtle Motion of Human Joints

ShadowDancXR: Body Gesture Digitization for Low-cost Extended Reality (XR) Headsets

Contact Info

Product

Resources

About