Multimodal Assessment of Parkinson's Disease: A Deep Learning Approach

Abstract

Parkinson's disease is a neurodegenerative disorder characterized by a variety of motor symptoms. Particularly, difficulties to start/stop movements have been observed in patients. From a technical/diagnostic point of view, these movement changes can be assessed by modeling the transitions between voiced and unvoiced segments in speech, the movement when the patient starts or stops a new stroke in handwriting, or the movement when the patient starts or stops the walking process. This study proposes a methodology to model such difficulties to start or to stop movements considering information from speech, handwriting, and gait. We used those transitions to train convolutional neural networks to classify patients and healthy subjects. The neurological state of the patients was also evaluated according to different stages of the disease (initial, intermediate, and advanced). In addition, we evaluated the robustness of the proposed approach when considering speech signals in three different languages: Spanish, German, and Czech. According to the results, the fusion of information from the three modalities is highly accurate to classify patients and healthy subjects, and it shows to be suitable to assess the neurological state of the patients in several stages of the disease. We also aimed to interpret the feature maps obtained from the deep learning architectures with respect to the presence or absence of the disease and the neurological state of the patients. As far as we know, this is one of the first works that considers multimodal information to assess Parkinson's disease following a deep learning approach.

Hybrid Feature Extraction for Detection of Degree of Motor Fluctuation Severity in Parkinson’s Disease Patients

“…Moreover, our algorithm detects changes in motor fluctuation severity regardless of the patterns of the concurring activity. This is while the current UPDRS III score detection methods mainly require the subjects to perform some specific activities, such as leg agility task [ 16 ]; finger tap and hand grasp actions used in the UPDRS examination [ 17 ]; leg agility, sit–stand, and gait tasks [ 18 ]; handwriting, speech, and gait [ 19 ]; finger tapping, diadochokinesia, and toe tapping [ 9 ]; and finger tapping, speech, and gait [ 20 , 35 ]. These approaches are not well suited for the medication adjustment application as they require the subjects’ to engage with the system continuously and actively to provide comprehensive assessment of the temporal variability of motor fluctuation.…”

“…However, they generate abstract indices about motor impairments (i.e., tremor and bradykinesia), which have not been associated with the degree of motor fluctuation severity that is required for the treating physician to effectively adjust therapy. Other approaches attempt to estimate UPDRS III scores from each time point based on some symptom-based features (e.g., spectral power in 4–6 Hz frequencies for tremor and 1–4 Hz frequencies for bradykinesia) [ 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] or deep learning algorithms [ 22 , 23 ]. Such approaches could be useful in detection of PD from healthy subjects.…”

Comparison of User Models Based on GMM-UBM and I-Vectors for Speech, Handwriting, and Gait Assessment of Parkinson’s Disease Patients

“…The subjects performed a total of 14 exercises divided into writing and drawing tasks. Additional information about the handwriting exercises can be found in [10]. Gait signals were captured with the eGaIT system, which consists of a 3D-accelerometer (range ±6g) and a 3D gyroscope (range ±500 • /s) attached to the external side (at the ankle level) of the shoes [25].…”

Feature Space Visualization with Spatial Similarity Maps for Pathological Speech Data

“…Additionally, we conducted a second evaluation with a data set collected from 85 participants, 37 of them diagnosed with Parkinson's Disease (PD). This is a subset of the database used in [18]. Within the PD group, there were 22 female and 15 male participants.…”