Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study

Abstract

Background Story recall is a simple and sensitive cognitive test that is commonly used to measure changes in episodic memory function in early Alzheimer disease (AD). Recent advances in digital technology and natural language processing methods make this test a candidate for automated administration and scoring. Multiple parallel test stimuli are required for higher-frequency disease monitoring. Objective This study aims to develop and validate a remote and fully automated story recall task, suitable for longitudinal assessment, in a population of older adults with and without mild cognitive impairment (MCI) or mild AD. Methods The “Amyloid Prediction in Early Stage Alzheimer’s disease” (AMYPRED) studies recruited participants in the United Kingdom (AMYPRED-UK: NCT04828122) and the United States (AMYPRED-US: NCT04928976). Participants were asked to complete optional daily self-administered assessments remotely on their smart devices over 7 to 8 days. Assessments included immediate and delayed recall of 3 stories from the Automatic Story Recall Task (ASRT), a test with multiple parallel stimuli (18 short stories and 18 long stories) balanced for key linguistic and discourse metrics. Verbal responses were recorded and securely transferred from participants’ personal devices and automatically transcribed and scored using text similarity metrics between the source text and retelling to derive a generalized match score. Group differences in adherence and task performance were examined using logistic and linear mixed models, respectively. Correlational analysis examined parallel-forms reliability of ASRTs and convergent validity with cognitive tests (Logical Memory Test and Preclinical Alzheimer’s Cognitive Composite with semantic processing). Acceptability and usability data were obtained using a remotely administered questionnaire. Results Of the 200 participants recruited in the AMYPRED studies, 151 (75.5%)—78 cognitively unimpaired (CU) and 73 MCI or mild AD—engaged in optional remote assessments. Adherence to daily assessment was moderate and did not decline over time but was higher in CU participants (ASRTs were completed each day by 73/106, 68.9% participants with MCI or mild AD and 78/94, 83% CU participants). Participants reported favorable task usability: infrequent technical problems, easy use of the app, and a broad interest in the tasks. Task performance improved modestly across the week and was better for immediate recall. The generalized match scores were lower in participants with MCI or mild AD (Cohen d=1.54). Parallel-forms reliability of ASRT stories was moderate to strong for immediate recall (mean rho 0.73, range 0.56-0.88) and delayed recall (mean rho=0.73, range=0.54-0.86). The ASRTs showed moderate convergent validity with established cognitive tests. Conclusions The unsupervised, self-administered ASRT task is sensitive to cognitive impairments in MCI and mild AD. The task showed good usability, high parallel-forms reliability, and high convergent validity with established cognitive tests. Remote, low-cost, low-burden, and automatically scored speech assessments could support diagnostic screening, health care, and treatment monitoring.

A remote speech‐based AI system to screen for early Alzheimer's disease via smartphones

“…Prior results show that task performance on the ASRT test, as measured by automated assessment of paraphrase‐enabled recall, correlates moderately with performance on the Logical Memory Delayed Recall test, a test commonly used to evaluate episodic memory function. 19 Similarly, the results for MCI from the current paper likely incorporate aspects of episodic memory (how much of the source text was retained and recollected), commonly impaired in MCI. However, the multifactorial output of the model allows it to simultaneously pick up on other differences in the text pairs.…”

“…At least one complete optional remote self‐assessment was completed by 86.5% (115/133, Figure 2 ). For those who engaged in at least one optional remote self‐assessment, overall engagement with daily testing was high (adherence to daily tests 78% in CU and 65% in MCI/mild AD 19 ). No adverse events were reported during self‐assessments.…”

“…Full details are provided in Skirrow et al. 19 In the current study, the index test (ASRT system) is derived from data collected during remote, smartphone‐based assessments.…”

“…A more detailed breakdown of story structure and balancing is provided in Skirrow et al. 19 ASRTs were administered in triplets (three stories administered consecutively each day). The self‐assessment schedule is provided in supplementary materials (Table S2 in supporting information).…”

“…Although adherence varied across participants, ASRTs have high parallel forms reliability, and only modest improvement on repeated administration of the ASRT task, indicating that different story triplets can be substituted for one another. 19 …”