Deep learning with an attention mechanism for continuous biomechanical motion estimation across varied activities

Abstract

Reliable estimation of desired motion trajectories plays a crucial part in the continuous control of lower extremity assistance devices such as prostheses and orthoses. Moreover, reliable estimation methods are also required to predict hard-to-measure biomechanical quantities (e.g., joint contact moment/force) for use in sports injury science. Recognising that human locomotion is an inherently time-sequential and limb-synergetic behaviour, this study investigates models and learning algorithms for predicting the motion of a subject’s leg from the motion of complementary limbs. The novel deep learning model architectures proposed are based on the Long Short-Term Memory approach with the addition of an attention mechanism. A dataset comprising Inertial Measurement Unit signals from 21 subjects traversing varied terrains was used, including stair ascent/descent, ramp ascent/descent, stopped, level-ground walking and the transitions between these conditions. Fourier Analysis is deployed to evaluate the model robustness, in addition to assessing time-based prediction errors. The experiment on three unseen test participants suggests that the branched neural network structure is preferred to tackle the multioutput problem, and the inclusion of an attention mechanism demonstrates improved performance in terms of accuracy, robustness and network size. An experimental comparison found that 57% of the model parameters were not needed after adding attention layers meanwhile the prediction error is lower than the LSTM model without attention mechanism. The attention model has errors of 9.06% and 7.64% (normalised root mean square error) for ankle and hip acceleration prediction respectively. Also, less high-frequency noise is present in the attention model predictions. We conclude that the internal structure of the proposed deep learning model is justified, principally the benefit of using an attention mechanism. Experimental results for biomechanical motion estimation are obtained, showing greater accuracy than only with LSTM. The trained attention model can be used throughout despite transitioning between terrain types. Such a model will be useful in, for example, the control of lower-limb prostheses, instead of the need to identify and switch between different trajectory generators for different walking modes.

A Deep Learning Model with a Self-Attention Mechanism for Leg Joint Angle Estimation across Varied Locomotion Modes

“…To overcome this, LSTM introduces a memory cell to deliver information across many timesteps. A gate mechanism is used to assist the memory cell in retaining crucial temporal information, while discarding useless information [ 26 ]. When processing the input in the timestep t, as shown in Figure 3 , the memory cell will first forget irrelevant information from via the forget gate and then be updated by new inputs and the last hidden state through the update gate.…”

“…In order to enhance model performance by better identifying critical inputs, several attention models [ 5 , 26 ] have been used in the estimation of the biomechanical parameters. Recently, the self-attention mechanism employed in the Transformer model [ 27 ] has been used for motion inference.…”

“…Prior to being fed into the model, the input data are normalized, which aids in the model training by eliminating the effect of different magnitudes across the features [2]. The normalization scaling factors (i.e., mean and standard deviation) obtained from the training dataset should be applied to the crossvalidation and testing datasets as well [26]. from the training dataset should be applied to the cross-validation and testing datasets as well [26].…”

“…The normalization scaling factors (i.e., mean and standard deviation) obtained from the training dataset should be applied to the crossvalidation and testing datasets as well [26]. from the training dataset should be applied to the cross-validation and testing datasets as well [26]. As for the dataset splitting, a 10-fold leave-one-out inter-subject testing scheme is implemented in order to maximally verify the model's generalization across all subjects.…”

“…In their study, Rai et al [ 5 ] adopted a dual-stage attention model to extract temporal features, with two LSTM layers acting as the encoder and decoder. In addition, Ding et al [ 26 ] proposed a more concise attention mechanism with a small multilayer perceptron to weight each timestep within the time series. With the prevalence of the Transformer model [ 27 ], the self-attention mechanism is increasingly being applied to a variety of time-series tasks, and some pilot studies have been conducted for human motion estimation as follows.…”