Feasibility of Digital Memory Assessments in an Unsupervised and Remote Study Setting

Abstract

Sensitive and frequent digital remote memory assessments via mobile devices hold the promise to facilitate the detection of cognitive impairment and decline. However, in order to be successful at scale, cognitive tests need to be applicable in unsupervised settings and confounding factors need to be understood. This study explored the feasibility of completely unsupervised digital cognitive assessments using three novel memory tasks in a Citizen Science project across Germany. To that end, the study aimed to identify factors associated with stronger participant retention, to examine test-retest reliability and the extent of practice effects, as well as to investigate the influence of uncontrolled settings such as time of day, delay between sessions or screen size on memory performance. A total of 1,407 adults (aged 18–89) participated in the study for up to 12 weeks, completing weekly memory tasks in addition to short questionnaires regarding sleep duration, subjective cognitive complaints as well as cold symptoms. Participation across memory tasks was pseudorandomized such that individuals were assigned to one of three memory paradigms resulting in three otherwise identical sub-studies. One hundred thirty-eight participants contributed to two of the three paradigms. Critically, for each memory task 12 independent parallel test sets were used to minimize effects of repeated testing. First, we observed a mean participant retention time of 44 days, or 4 active test sessions, and 77.5% compliance to the study protocol in an unsupervised setting with no contact between participants and study personnel, payment or feedback. We identified subject-level factors that contributed to higher retention times. Second, we found minor practice effects associated with repeated cognitive testing, and reveal evidence for acceptable-to-good retest reliability of mobile testing. Third, we show that memory performance assessed through repeated digital assessments was strongly associated with age in all paradigms, and individuals with subjectively reported cognitive decline presented lower mnemonic discrimination accuracy compared to non-complaining participants. Finally, we identified design-related factors that need to be incorporated in future studies such as the time delay between test sessions. Our results demonstrate the feasibility of fully unsupervised digital remote memory assessments and identify critical factors to account for in future studies.

Unsupervised mobile app-based cognitive testing in a population-based study of older adults born 1944

“…In the course of the follow-up in-clinic examination, 692 of 879 participants have thus far been invited to participate in a two-year study using the neotiv-App ( 12 , 34 ). Of these, 172 participants have signed the informed consent and initiated the mobile app-based cognitive testing to date.…”

“…In ORR-LDR, participants see a computer-generated room with two 3D-rendered objects (see Figure 2 ) ( 12 , 34 ). In the encoding phase, participants learn the position of two objects in a room and are tested for immediate retrieval.…”

“…Mobile app-based cognitive testing can address some of the shortcomings of conventional in-clinic cognitive testing. For example, automatically alternating test versions avoid or diminish practice effects ( 11 , 12 ) and enable more frequent and flexible cognitive testing ( 13 ). More frequent assessments may also increase reliability in cognitive measurement ( 14 , 15 ).…”

Priorities for research on neuromodulatory subcortical systems in Alzheimer's disease: Position paper from the NSS PIA of ISTAART

“…Currently, psychometric assessment tools for individuals with AD, such as the Neuropsychiatric Inventory (NPI), 97 rely on retrospective subjective ratings by proxy, which are vulnerable to recall bias in quasi‐experimental clinical settings. Digital instruments can assess affect, sleep, vigilance, agitation, social behavior, and psychiatric manifestations via objective and immediate measures in ecologically valid (e.g., in‐home) settings 98,99 . Furthermore, more sensitive, ecologically valid neuropsychiatric instruments may offer advantages for monitoring the success of disease‐modifying therapeutics compared to evaluating cognitive changes alone.…”

Web-based cognitive assessment in older adults: Where do we stand?

“…However, counterbalanced within-person testing has demonstrated that the testing environment does not significantly impact the scores of older adults from unsupervised at-home testing compared to supervised lab testing [24]. Within-person studies also show good testretest reliability [25,26,27 ] and poor to good convergent validity compared to analogous neuropsychological tasks [24,25, & ]. This suggests excellent construct validity, although with slightly lower scores on fine motor or fast reaction time tests due to differences in device latencies and interfaces [19,32].…”