A Novel Approach for Upper Limb Functionality Assessment Based on Deep Learning and Multimodal Sensing Data

Abstract

Upper limb rehabilitation is an effective methodology to restore and improve the functionality of patients after multiple medical events, such as strokes, arthroscopic surgery, and breast cancer surgery. High-quality rehabilitation training can promote the independent living of patients, thus enhancing the quality of life and reducing the financial burden. Traditional training sessions have some limitations, including high expenses, low compliance, and inaccurate evaluations. This paper presents a novel approach to assist healthcare professionals in assessing the functionality of upper limbs based on multimodal sensing data and deep learning algorithms. There are five different types of sensing data employed in the proposed approach: accelerometer, angular velocity, device orientation, RGB image, and depth image data. In order to assess the accuracy of training actions, the presented approach applies two machine learning algorithms, which are the dynamic time warping-K-nearest neighbor (DTW-KNN) algorithm and the long short-term memory (LSTM) neural network. The experimental results show that multimodal sensing data can improve the modeling accuracy compared with unimodal sensing data. The LSTM model can achieve better accuracy (96.3%) than DTW-KNN (74.07%) with multimodal sensing data. Moreover, LSTM performs extremely efficiently in modeling high-dimensional sensing data.

Assessing Human Motion During Exercise Using Machine Learning: A Literature Review

“…Fifty-four publications used optical-based systems and extracted a full-body human skeleton from the captured data. Thirty-six out of the 54 publications used as features only the position of the joints, while six publications used the orientation of the joints [50], [58], [66], [70], [78], [83]. Another six publications used features of both modalities together [45], [55], [56], [85], [96], [100].…”

“…The other big category of algorithms used under discriminative models was Recurrent Neural Networks (RNN), which were used in 11/31 publications. Long Short-Term Memory (LSTM) RNN, in particular, were used in nine publications, making it the most used ML algorithm in the publications [44], [51], [74], [81], [83], [85], [101], [106], [115]. A traditional RNN [102], and a Gated Recurrent Unity Network (GRU) [45] were used in one publication each.…”

“…Of the ones that used classification, most (21/88) used binary classification [36], [39], [49], [55], [59], [60], [61], [65], [66], [71], [72], [79], [84], [101], [102], [105], [106], [111], [114], [117], [120]; that is, datasets were assessed using a dichotomous variable (e.g., normal or abnormal motion). Other often-used types of assessments were using 3-point ordinal scale values evaluated from rank-based classification approaches [35], [48], [54], [62], [63], [64], [68], [77], [83], [86], [87], [97], [98], [99], [103], [107], [108], [109], and score-based assessments evaluated using regression methods [31], [40], [44], [45], [46], [47], [56],…”

“…In addition to the above metrics, nine publications reported the time required for a trained model to assess new data [3], [34], [46], [83], [90], [101], [102], [111], [118]. These times ranged from a few milliseconds to several seconds.…”

“…In the former of the two, Ross et al [109], however, also noted that despite having higher accuracies, feedback generated from the inertial data was harder to interpret and provide meaningful feedback that could lead to improvements during exercise. In addition, the fusion of data from both technologies has been used in some publications [50], [58], [79], [83], [102]. A comparison between the use of fused data and separate input modalities has shown higher accuracy from algorithms trained using fused data [79], [173].…”