Objective To validate a fully automated scoring algorithm for the Rey-Osterrieth Complex Figure Test (ROCFT) by comparing the scoring results of the algorithm to the results of human raters. Method The algorithm consisted of a cascade of deep neural networks which were trained on human rater scores to extract the 18 segments of the figure, and to quantify the patient’s performance. Algorithm results were compared to six expert raters for 303 drawings. We tested whether the average correlation between algorithm scores and scores by all human raters was equivalent to the average inter-rater correlation (with equality bound Δr < .05). The immediate and delayed recall trial were used; the copy trial showed a strong ceiling effect. Results The mean Pearson correlation between raters was .94 (SD = 0.01). The correlation between to algorithm and the raters was .88 (SD = 0.02). A two-one-sided t-tests (TOST) equivalence test showed that these correlations were not strictly equivalent, t(5) = 4.02, p = .995, 95% CI [0.35, 0.52]. Conclusions Although not strictly equivalent to human ratings, the algorithm’s performance is high, approaching a level of reliability found among human raters. We expect that improved individual segment detection will bring the algorithm scoring accuracy on par with that of human raters. Algorithmic scoring of the ROCFT will likely save valuable time and lead to higher levels of standardization in clinical practice.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.