Use of Patient-Reported Symptoms from an Online Symptom Tracking Tool for Dementia Severity Staging: Development and Validation of a Machine Learning Approach

Abstract

Background SymptomGuide Dementia (DGI Clinical Inc) is a publicly available online symptom tracking tool to support caregivers of persons living with dementia. The value of such data are enhanced when the specific dementia stage is identified. Objective We aimed to develop a supervised machine learning algorithm to classify dementia stages based on tracked symptoms. Methods We employed clinical data from 717 people from 3 sources: (1) a memory clinic; (2) long-term care; and (3) an open-label trial of donepezil in vascular and mixed dementia (VASPECT). Symptoms were captured with SymptomGuide Dementia. A clinician classified participants into 4 groups using either the Functional Assessment Staging Test or the Global Deterioration Scale as mild cognitive impairment, mild dementia, moderate dementia, or severe dementia. Individualized symptom profiles from the pooled data were used to train machine learning models to predict dementia severity. Models trained with 6 different machine learning algorithms were compared using nested cross-validation to identify the best performing model. Model performance was assessed using measures of balanced accuracy, precision, recall, Cohen κ, area under the receiver operating characteristic curve (AUROC), and area under the precision-recall curve (AUPRC). The best performing algorithm was used to train a model optimized for balanced accuracy. Results The study population was mostly female (424/717, 59.1%), older adults (mean 77.3 years, SD 10.6, range 40-100) with mild to moderate dementia (332/717, 46.3%). Age, duration of symptoms, 37 unique dementia symptoms, and 10 symptom-derived variables were used to distinguish dementia stages. A model trained with a support vector machine learning algorithm using a one-versus-rest approach showed the best performance. The correct dementia stage was identified with 83% balanced accuracy (Cohen κ=0.81, AUPRC 0.91, AUROC 0.96). The best performance was seen when classifying severe dementia (AUROC 0.99). Conclusions A supervised machine learning algorithm exhibited excellent performance in identifying dementia stages based on dementia symptoms reported in an online environment. This novel dementia staging algorithm can be used to describe dementia stage based on user-reported symptoms. This type of symptom recording offers real-world data that reflect important symptoms in people with dementia.

Patterns of Symptom Tracking by Caregivers and Patients With Dementia and Mild Cognitive Impairment: Cross-sectional Study

“…Furthermore, as the myGoalNav is not a checklist of symptoms, the tracking frequencies presented here are distinct from symptom prevalence . There are also limitations in the development of the staging algorithm model, such as potential bias in the training data because of clinician facilitation [ 44 ].…”

“…In 2013, we developed an artificial neural network model to stage dementia, which was trained on data from 320 memory clinic patients [ 42 ]. That model was updated in 2020 using a support vector machine–supervised learning algorithm trained on 717 patients [ 44 ]. Data from these patients were captured with myGoalNav in a memory clinic, a long-term care study [ 45 ], and a dementia clinical trial [ 42 ].…”

“…Patient age and their symptom profiles served as inputs to train the model to predict the dementia stage. Further details of the model, including algorithm choice and performance, can be found in the study by Shehzad et al [ 44 ].…”

Applied machine learning in Alzheimer's disease research: omics, imaging, and clinical data

“…Vizcarra et al [ 112 ] validated previously published ML algorithms using convolutional neural networks (CNNs) and to determine if pathological heterogeneity may alter algorithm-derived measures using 40 cases from the Goizueta Emory Alzheimer's Disease Center brain bank, which displays an array of pathological diagnoses (including AD with and without Lewy body disease (LBD) and/or TDP-43-positive inclusions) and evaluated their levels of Aβ pathologies. Shehzad et al [ 113 ] used individualized symptom profiles from the pooled data (clinical data from 717 people from three sources: (1) a memory clinic, (2) long-term care, and (3) an open-label trial of donepezil in vascular and mixed dementia) to train various ML models to predict dementia severity (MCI, mild dementia, moderate dementia, or severe dementia). Tsao et al [ 114 ] combined a predictive multi-task ML method (cFSGL) with a novel ML-based multivariate morphometric surface map of the hippocampus (mTBM) to predict future cognitive scores (Alzheimer's Disease Assessment Scale cognitive scores 6, 12, 24, 36, and 48 months from baseline) of patients.…”

Random Forest Model in the Diagnosis of Dementia Patients with Normal Mini-Mental State Examination Scores

“…Recent studies had shown that machine learning (ML) exhibited excellent performance in identifying MCI and dementia [11][12][13][14][15][16][17], but these mostly used biomarker data such as neuroimaging and CSF components that were expensive technologies [12,13,16]. ML diagnostic models based on cognitive data were gradually being applied [11,15,18,19].…”