Estimation of kinematics from inertial measurement units using a combined deep learning and optimization framework

Abstract

The difficulty of estimating joint kinematics remains a critical barrier toward widespread use of inertial measurement units in biomechanics. Traditional sensor-fusion filters are largely reliant on magnetometer readings, which may be disturbed in uncontrolled environments. Careful sensor-to-segment alignment and calibration strategies are also necessary, which may burden users and lead to further error in uncontrolled settings. We introduce a new framework that combines deep learning and top-down optimization to accurately predict lower extremity joint angles directly from inertial data, without relying on magnetometer readings. We trained deep neural networks on a large set of synthetic inertial data derived from a clinical marker-based motion-tracking database of hundreds of subjects. We used data augmentation techniques and an automated calibration approach to reduce error due to variability in sensor placement and limb alignment. On leftout subjects, lower extremity kinematics could be predicted with a mean (± STD) root mean squared error of less than 1.27 • (± 0.38 • ) in flexion/extension, less than 2.52 • (± 0.98 • ) in ad/abduction, and less than 3.34 • (± 1.02 • ) internal/external rotation, across walking and running trials. Errors decreased exponentially with the amount of training data, confirming the need for large datasets when training deep neural networks. While this framework remains to be validated with true inertial measurement unit (IMU) data, the results presented here are a promising advance toward convenient estimation of gait kinematics in natural environments. Progress in this direction could enable large-scale studies and offer an unprecedented view into disease progression, patient recovery, and sports biomechanics.

A Comparison of Three Neural Network Approaches for Estimating Joint Angles and Moments from Inertial Measurement Units

“…A variety of methods have been proposed for this purpose, and reviewed by Ancillao et al [ 9 ]. To overcome accuracy limitations and the restricted subsets of parameters that can be determined, researchers have focused on applying machine learning methods to improve the prediction of GRFs, joint angles and joint moments [ 2 , 10 , 11 , 12 , 13 , 14 , 15 ], with initial efforts focused on predicting smaller subsets of data, such as single GRF and joint moment components [ 10 , 11 , 12 ], or in the case of Stetter et al [ 13 ] by predicting sagittal and frontal plane moments in isolation. Very recently gait researchers have trained machine learning models to predict all component joint angles [ 14 , 15 ] and moments across all lower limb joints [ 14 ].…”

“…They work especially well on data with a spatial relationship, and due to the fact an ordered relationship can also be found in time series data, this makes CNNs suitable for time series prediction of human motion. CNNs have been used with inertial sensor data inputs to predict joint kinematics and kinetics [ 11 , 15 ]. Different open-access models have been trained on large datasets for image classification previously, enabling the use of transfer learning or fine tuning of a model instead of training a CNN from scratch.…”

“…Unfortunately, LSTM networks are intensive to train and require large datasets, something rarely available in biomechanics [ 2 ]. Using inertial sensor data as inputs, LSTM networks have also been previously used to predict joint angles and moments [ 15 , 17 , 20 ].…”

DeepBBWAE-Net: A CNN-RNN Based Deep SuperLearner For Estimating Lower Extremity Sagittal Plane Joint Kinematics Using Shoe-Mounted IMU Sensors In Daily Living

“…In [18], five IMU sensors were used to estimate kinematics in walking and running on a treadmill. More recently, in [19], a combined deep learning and optimization framework was proposed to estimate 3D kinematics using seven IMU sensors for walking and running. These studies have used a relatively large set of sensors and were only conducted on treadmill or overground conditions.…”

“…Many studies have employed machine learning for kinematics estimation of treadmill and overground walking conditions [13], [15], [16], [18], [17], [19] using a large number of sensors. Due to the impracticality associated with using a full set of IMU sensors, a limited number of studies have employed a reduced set of IMU sensors to estimate joint kinematics.…”