Deep Learning-Based Upper Limb Functional Assessment Using a Single Kinect v2 Sensor

Abstract

We develop a deep learning refined kinematic model for accurately assessing upper limb joint angles using a single Kinect v2 sensor. We train a long short-term memory recurrent neural network using a supervised machine learning architecture to compensate for the systematic error of the Kinect kinematic model, taking a marker-based three-dimensional motion capture system (3DMC) as the golden standard. A series of upper limb functional task experiments were conducted, namely hand to the contralateral shoulder, hand to mouth or drinking, combing hair, and hand to back pocket. Our deep learning-based model significantly improves the performance of a single Kinect v2 sensor for all investigated upper limb joint angles across all functional tasks. Using a single Kinect v2 sensor, our deep learning-based model could measure shoulder and elbow flexion/extension waveforms with mean CMCs >0.93 for all tasks, shoulder adduction/abduction, and internal/external rotation waveforms with mean CMCs >0.8 for most of the tasks. The mean deviations of angles at the point of target achieved and range of motion are under 5° for all investigated joint angles during all functional tasks. Compared with the 3DMC, our presented system is easier to operate and needs less laboratory space.

Placement Recommendations for Single Kinect-Based Motion Capture System in Unilateral Dynamic Motion Analysis

“…We carried out a series of residual analyses on the raw data using the cut-off frequencies of 4, 5, 6, 7, and 8 Hz, respectively. We selected 6 Hz as the cut-off frequency because it yields the best result in our task and is validated for upper limb function assessment [12]. Local segment coordination (LSC) systems of the torso and upper arm (taking the right arm as an example) are defined in Table 1.…”

“…Kinect SDK features real-time skeletal tracking of 3D locations for skeletal joints, with its RGB-D sensor and human pose estimation algorithm [9]. Such low-cost, portable Kinect sensors achieved great popularity in motion analysis [10][11][12][13].…”

“…Single Kinect-based systems were also used in gait analysis [7,11,[16][17][18], gait rehabilitation training [19][20][21], trunk and lower limb kinematics during dynamic balance test [10], or squatting [22] and jumping tasks [23]. The systems were also used in the kinematic assessment or rehabilitation training of upper limb functional tasks for healthy people [12] and for patients with motor disorders [22,[24][25][26][27]. The single Kinect-based systems generally have good reliability in both spatial and temporal parameters and have better capability in assessing temporal parameters compared with spatial parameters [7,11,12,16,17].…”

“…The systems were also used in the kinematic assessment or rehabilitation training of upper limb functional tasks for healthy people [12] and for patients with motor disorders [22,[24][25][26][27]. The single Kinect-based systems generally have good reliability in both spatial and temporal parameters and have better capability in assessing temporal parameters compared with spatial parameters [7,11,12,16,17]. Compared with the static pose assessment, performing dynamic motion analysis using a single Kinect-based system is more challenging because of motion artifacts [28].…”

Detection of Postural Control in Young and Elderly Adults Using Deep and Machine Learning Methods with Joint–Node Plots

“…One study investigated the extent to which such deep learning–based systems provide satisfactory accuracy in exergame-relevant measures; a deep learning–based system was reported to perform as well as the gold standard system in the detection of temporal variations [ 38 ]. In one study, a long short-term memory recurrent neural network was used in a supervised machine learning architecture and a novel deep learning–refined kinematic model with good kinematic accuracy for upper limb functional assessment was developed [ 39 ]. Therefore, Kinect’s image information combined with machine and deep learning can be used to develop an effective limb functional assessment system for medical diagnosis or therapeutic evaluation.…”