Automating Rey Complex Figure Test scoring using a deep learning-based approach: A potential large-scale screening tool for congnitive decline

Abstract

Background: The Rey Complex Figure Test (RCFT) has been widely used to evaluate neurocognitive functions in various clinical groups with a broad range of ages. However, despite its usefulness, the scoring method is as complex as the figure. Such a complicated scoring system can lead to the risk of reducing the extent of agreement among raters. Although several attempts have been made to use RCFT in clinical settings in a digitalized format, little attention has been given to develop direct automatic scoring that is comparable to experienced psychologists. Therefore, we aimed to develop an artificial intelligence (AI) scoring system for RCFT using a deep learning (DL) algorithm and confirmed its validity.Methods:A total of 6,680 subjects were enrolled in the Gwangju Alzheimer’s and Related Dementia cohort registry, Korea from January 2015 to June 2021. We obtained 20,040 scanned images using three images per subject (copy, immediate recall, and delayed recall) and scores rated by 32 experienced psychologists. We trained the automated scoring system using the DenseNet architecture. To increase the model performance, we improved the quality of training data by re-examining some images with poor results (mean absolute error (MAE) 5 [points]) and re-trained our model. Finally, we conducted an external validation with 150 images scored by five experienced psychologists.Results: For five-fold cross-validation, our first model obtained MAE = 1.24 [points] and R-squared ( ) = 0.977. However, after evaluating and updating the model, the performance of the final model was improved (MAE = 0.95 [points], = 0.986). Predicted scores among cognitively normal, mild cognitive impairment, and dementia were significantly differed. For the 150 independent test sets, the MAE and between AI and average scores by five human experts was 0.64 [points] and 0.994, respectively.Conclusion: We concluded that there was no fundamental difference between the rating scores of experienced psychologists and those of our AI scoring system. We expect that our AI psychologist will be able to contribute to screen the early stages of Alzheimer’s disease pathology in medical checkup centers or large-scale community-based research institutes in a faster and cost-effective way.

Development of an eye-tracking system based on a deep learning model to assess executive function in patients with mental illnesses

“…The model also offered a rapid and simple measure of impaired executive function, taking only 3 minutes from administration to assessment through computerized eye movement recording and deep learning analysis. This process is considerably faster and simpler than the traditional RCFT scoring system and the automated scoring systems reported in previous studies, which still require a prior drawing process [20][21][22][23] . This improved model provides bene ts and is easy to apply in real-world clinical and research settings, saving a signi cant amount of labor and time and reducing human scoring variability.…”

“…Considerable efforts have been made to overcome the limitations of the RCFT scoring system, including the development of an automated scoring system using photos of RCFT drawings and a deep learning algorithm [20][21][22][23] , the implementation of a tablet-based digital drawing assessment 24 , and the adoption of a simpler method for scoring organizational strategy (0 or 1 points) 25 . Although these approaches have made substantial advancements in addressing the complexity, labor intensity, and scoring variability of the scoring system, there are still limitations in its administration given that it is a time-consuming, visuomotor function-affected, indirect drawing test.…”

Interpretable deep learning approach for extracting cognitive features from hand-drawn images of intersecting pentagons in older adults

“…Automated scoring methods that utilize machine learning methods offer the potential to address some or all of these weaknesses. For example, deep-learning techniques have demonstrated promising performance in automating ratings for the PDT 7,8 , Rey Complex Figure Test 9,10 , and Clock Drawing Test 11,12 . However, the primary objective of the previous automated scoring is to reproduce human-based conventional ratings and few machine learning approaches directly predict cognitive performance from drawings 13 .…”