Automated scoring of the Rey-Osterrieth Complex Figure Test using a deep-learning algorithm

Abstract

Objective To validate a fully automated scoring algorithm for the Rey-Osterrieth Complex Figure Test (ROCFT) by comparing the scoring results of the algorithm to the results of human raters. Method The algorithm consisted of a cascade of deep neural networks which were trained on human rater scores to extract the 18 segments of the figure, and to quantify the patient’s performance. Algorithm results were compared to six expert raters for 303 drawings. We tested whether the average correlation between algorithm scores and scores by all human raters was equivalent to the average inter-rater correlation (with equality bound Δr < .05). The immediate and delayed recall trial were used; the copy trial showed a strong ceiling effect. Results The mean Pearson correlation between raters was .94 (SD = 0.01). The correlation between to algorithm and the raters was .88 (SD = 0.02). A two-one-sided t-tests (TOST) equivalence test showed that these correlations were not strictly equivalent, t(5) = 4.02, p = .995, 95% CI [0.35, 0.52]. Conclusions Although not strictly equivalent to human ratings, the algorithm’s performance is high, approaching a level of reliability found among human raters. We expect that improved individual segment detection will bring the algorithm scoring accuracy on par with that of human raters. Algorithmic scoring of the ROCFT will likely save valuable time and lead to higher levels of standardization in clinical practice.

Overview of the Complex Figure Test and Its Clinical Application in Neuropsychiatric Disorders, Including Copying and Recall

“…At the same time, combined with machine learning algorithms, it can build classification and diagnosis models of different diseases. A recent study used the deep learning algorithm to automate ROCF scoring, which has high performance and is close to the reliability of human raters ( 78 ). Future research should use machine learning technology to provide information about the neuropsychological structure of different neuropsychiatric diseases and gradually establish a new digital scoring method to standardize clinical practice, such as a digital ROCF feature selection analysis algorithm.…”

Comparison of RCF Scoring System to Clinical Decision for the Rey Complex Figure Using Machine-Learning Algorithm

“…Poorer copying of a figure by a given patient in comparison with that by normal healthy controls could suggest Alzheimer's disease (AD). 7 It has been reported that RCF is one of neuropsychological tests with adequate sensitivity and specificity for discriminating patients with different neuropsychological conditions such as dementia, MCI, depression, and mild head injury. 8 Many clinicians have used RCF for diagnosing MCI and dementia to improve their clinical decisions.…”

“… 14 Other studies have reported a fully automated scoring algorithm for RCF by comparing scoring results of the algorithm to results of manual scoring. 7 The algorithm consisting of a cascade machine learning model was trained on manual scores to extract 18 segments of the RCF. The algorithm performance was high, but not strictly equivalent to manual scoring.…”

Automatic, Qualitative Scoring of the Interlocking Pentagon Drawing Test (PDT) Based on U-Net and Mobile Sensor Data

“…However, these quantify the accuracy of the drawings by hand and in a subjective manner due to the complexity of the figures [23]. To address this, an automated scoring algorithm for the ROCF test was recently suggested and is based on cascaded deep neural networks, trained on scores from human expert raters [24].…”