Estimation of Kinematics from Inertial Measurement Units Using a Combined Deep Learning and Optimization Framework

Abstract

The difficulty of estimating joint kinematics remains a critical barrier toward widespread use of inertial measurement units in biomechanics. Traditional sensor-fusion filters are largely reliant on magnetometer readings, which may be disturbed in uncontrolled environments. Careful sensor-to-segment alignment and calibration strategies are also necessary, which may burden users and lead to further error in uncontrolled settings. We introduce a new framework that combines deep learning and top-down optimization to accurately predict lower extremity joint angles directly from inertial data, without relying on magnetometer readings. We trained deep neural networks on a large set of synthetic inertial data derived from a clinical marker-based motion-tracking database of hundreds of subjects. We used data augmentation techniques and an automated calibration approach to reduce error due to variability in sensor placement and limb alignment. On left-out subjects, lower extremity kinematics could be predicted with a mean (± STD) root mean squared error of less than 1.27 ° (± 0.38 °) in flexion/extension, less than 2.52 ° (± 0.98 °) in ad/abduction, and less than 3.34 ° (± 1.02 °) internal/external rotation, across walking and running trials. Errors decreased exponentially with the amount of training data, confirming the need for large datasets when training deep neural networks. While this framework remains to be validated with true inertial measurement unit (IMU) data, the results presented here are a promising advance toward convenient estimation of gait kinematics in natural environments. Progress in this direction could enable large-scale studies and offer an unprecedented view into disease progression, patient recovery, and sports biomechanics.

DeepBBWAE-Net: A CNN-RNN Based Deep SuperLearner for Estimating Lower Extremity Sagittal Plane Joint Kinematics Using Shoe-Mounted IMU Sensors in Daily Living

“…In [24], five IMU sensors were used to estimate kinematics in walking and running on a treadmill using a deep model based on convolutional and long-short term memory recurrent layers (DeepConvLSTM) with mean absolute errors ranging from 2.2°to 5.1°. More recently, in [25], the authors combined deep learning and optimization frameworks to estimate 3D kinematics using seven IMU sensors for walking and running, achieving an RMSE of 1.27°( ± 0.38°) for knee flexion/extension. However, because they only validated their study with simulated IMU data, it is unclear how their method will perform using real IMU data.…”

DeepBBWAE-Net: A CNN-RNN Based Deep SuperLearner For Estimating Lower Extremity Sagittal Plane Joint Kinematics Using Shoe-Mounted IMU Sensors In Daily Living

“…For both RMSE and correlation, the base and intermediate model ranges were wider in the slope and stair conditions but more narrow for the treadmill and overground conditions. 19.25%, and 17.72% for the same conditions. In terms of Pearson correlation coefficients, the intermediate models saw an increase in correlation of 0.41%, 0.54%, 0.78%, 1.18%, 0.90%, and 1.20% while DeepBBWAE-Net had an increase of 0.82%, 1.19%, 1.89%, 2.62%, 1.89%, and 2.78% for the six walking conditions.…”

“…In [18], five IMU sensors were used to estimate kinematics in walking and running on a treadmill. More recently, in [19], a combined deep learning and optimization framework was proposed to estimate 3D kinematics using seven IMU sensors for walking and running. These studies have used a relatively large set of sensors and were only conducted on treadmill or overground conditions.…”

“…Many studies have employed machine learning for kinematics estimation of treadmill and overground walking conditions [13], [15], [16], [18], [17], [19] using a large number of sensors. Due to the impracticality associated with using a full set of IMU sensors, a limited number of studies have employed a reduced set of IMU sensors to estimate joint kinematics.…”

The Use of Synthetic IMU Signals in the Training of Deep Learning Models Significantly Improves the Accuracy of Joint Kinematic Predictions

“…While both studies used an LSTM model architecture, the current study required significantly fewer experimental observations to achieve an equivalent level of prediction accuracy. Rapp et al predicted hip and knee joint angles during gait in 420 subjects using an LSTM model coupled with an optimization algorithm to account for differences in the predicted and measured segment rotational velocities [ 42 ]. When evaluated on simulated IMU data, the combined algorithm achieved an RMSE of 4.2° at the knee and 4.1° at the hip prior to a calibration step, which further improved the accuracy.…”