Deep Learning-Based Multimodal Abnormal Gait Classification Using a 3D Skeleton and Plantar Foot Pressure

Abstract

Classification of pathological gaits has an important role in finding a weakened body part or disease and supporting a doctor's decision. Many machine learning-based approaches have been proposed that automatically classify abnormal gait patterns using various sensors, such as inertial sensors, depth cameras and foot pressure plates. In this paper, we present a deep learning-based abnormal gait classification method employing both a 3D skeleton (obtained with a depth camera) and plantar foot pressure. We collected skeleton and foot pressure data simultaneously for 1 normal and 5 pathological gaits (antalgic, lurching, steppage, stiff-legged, and Trendelenburg gaits) and classified the gaits by using single-modal models fed either skeleton or pressure data and a multimodal model fed both data types together. In the proposed method, we fed the sequential skeleton and average foot pressure data into recurrent neural network (RNN)-based encoding layers and convolutional neural network (CNN)-based encoding layers, respectively. Finally, the output features were concatenated and fed to the classification layers. The pressure-based and skeleton-based single-modal models achieved classification accuracies of 68.82% and 93.40%, respectively. The proposed multimodal hybrid model using skeleton and foot pressure data together showed improved performance, with an accuracy of 95.66%. We fine-tuned the hybrid model by applying a 3-step training methodology and ultimately increased the accuracy to 97.60%. This study indicates that the integrated features of the skeleton and foot pressure data represent both the spatiotemporal motion information and weight distribution, so data fusion can generate a positive effect in pathological gait classification.

Hybrid Optimized GRU-ECNN Models for Gait Recognition with Wearable IOT Devices

“…e GRU network consists of two gates and is considered faster than the LSTM and RNN models [43]. Where x t is the input feature at the current state, y t is the output state, h t is the output of the module at the current instant, Z t and r t are update and reset gates, W(t) is weights, and B(t) is bias weights at current instant.…”

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

“…In total, 60 articles were found related to the domain agnostic domain, where, 16 were in classification [170]- [185], 9 were in detection [186]- [194], 11 were in analysis [195]- [205], 8 were in recognition [206]- [213], 9 were in prediction [214]- [222], 1 was in language processing [223], 2 were in image processing [224], [225], 1 was in image retrieval [47], 2 were in integration [226], [227] and 1 was in segmentation [228]. Of 43 articles in the human activity domain, 20 were in recognition [229]- [248], 8 were in detection [249]- [256], 5 were in classification [257]- [261], 4 were in analysis [262]- [265], 3 were in identification [266]- [268] and 1 was in comparison [269], monitoring [270] and assessment [271]. In the emotion recognition domain, 28 articles are encountered; from them, 21 were in recognition [272]- [292], 2 were in analysis [293], [294], and prediction [295], [296], and 1 was in detection [297], assessment [298], and estimating [299].…”

Smart Cushion Based on Pressure Sensor Array for Human Sitting Posture Recognition

“…Chen et al [38] explored DL algorithms including ResNet50, InceptionV3, and MobileNet to identify differences in the response of walking speed to plantar pressure. Jun et al [39] performed pathological gaits classification, feeding the sequential skeleton and average foot pressure data into a recurrent neural network (RNN) based encoding layers and CNN-based encoding layers, respectively. The method effectively extracted features, then output features were connected and fed to a fully connected layer for classification.…”