Environmental Distractions during Unsupervised Remote Digital Cognitive Assessment

Abstract

The current demand for cognitive assessment cannot be met with traditional in-person methods, warranting the need for remote unsupervised options. However, lack of visibility into testing conditions and effort levels limit the utility of existing remote options. This retrospective study analyzed the frequency of and factors associated with environmental distractions during a brief digital assessment taken at home by 1,442 adults aged 23-84. Automated scoring algorithms flagged low data capture. Frequency of environmental distractions were manually counted on a per-frame and per-trial basis. A total of 7.4% of test administrations included distractions. Distractions were more frequent in men (41:350) than women (65:1,092) and the average age of distracted participants (51.7) was lower than undistracted participants (57.8). These results underscore the challenges associated with unsupervised cognitive assessment. Data collection methods that enable review of testing conditions are needed to confirm quality, usability, and actionability.

A novel computer adaptive word list memory test optimized for remote assessment: Psychometric properties and associations with neurodegenerative biomarkers in older women without dementia

“…We have previously observed lower performance on other self‐administered cognitive measures at home versus in clinic 30 . Madero et al 31 . similarly reported the presence of distraction in a minority of remote cognitive assessment sessions (7%), which had a negative impact on performance.…”

“…We have previously observed lower performance on other self-administered cognitive measures at home versus in clinic. 30 Madero et al 31 similarly reported the presence of distraction in a minority of remote cognitive assessment sessions (7%), which had a negative impact on performance. Future work will examine whether participant self-report of interference can help reduce variability introduced by testing in an unsupervised environment.…”

The Geneva Space Cruiser: a fully self-administered online tool to assess prospective memory across the adult lifespan

“…Since the initial launch of the first online experiments, multiple studies have suggested that computerised online assessment can provide results that are reliable and comparable to laboratory assessment across different cognitive domains (such as working memory, executive functioning, episodic memory, verbal fluency, attention, vigilance, or complex psycholinguistic abilities), and even for full-scale neuropsychological test batteries such as the Cambridge Neuropsychological Test Automated Battery or the Cognitive Function Test (Backx et al, 2020;Feenstra et al, 2017;Kim et al, 2019;Luna et al, 2020;Trustram Eve & de Jager, 2014). However, certain studies have also highlighted that performance significantly varies between the two settings, that participants are more distracted during self-assessment, and that currently most online batteries lack normative data (Al Baghal, 2019;Gates & Kochan, 2015;Madero et al, 2021). This underlines the need for more research on the utility, comparability, and reliability of online assessments.…”

Stricker Learning Span criterion validity: a remote self-administered multi-device compatible digital word list memory measure shows similar ability to differentiate amyloid and tau PET-defined biomarker groups as in-person Auditory Verbal Learning Test

“…If cognitive screening / risk for cognitive decline is the goal, worse performance in remote settings may help identify risk in a way not captured by controlled clinical settings, adding an element of ecological validity in terms of ability to adapt to a new task without assistance. Future work will examine whether and to what degree these factors may influence test performance, as increased distractions in the home environment can negatively impact performance 58 . Finally, while use of a strictly biomarker-defined ground truth is a novel aspect of this study, in vivo PET biomarkers also have some limitations such as high but imperfect reliability (manifested by "noise" in the trajectories of imaging results over time in some individuals), and the fact that PET measures of amyloid and tau pathology have a sensitivity floor and medically significant pathology can exist that lies beneath this detection threshold 59 .…”