Test–Retest Reliability of Remote ImPACT Administration

Netzel, Lauren; Moran, Ryan; Hopfe, Dustin; Salvatore, Anthony P.; Brown, Warren; Murray, Nicholas G.

doi:10.1093/arclin/acab055

Cited by 4 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In 2 cohorts, we found smartphone cognitive tests were reliable within a single administration (ie, internally consistent) and across repeated assessments (ie, test-retest reliability) with no apparent differences by operating system. For all measures except go/no-go, reliability estimates were moderate to excellent and on par with other remote digital assessments 5 , 6 , 10 , 37 , 38 and in-clinic criterion standards. 39 , 40 , 41 Go/no-go showed similar within- and between-person variability in participants without symptoms (ie, poor reliability), and participant feedback suggested instructions were confusing and the stimuli disappeared too quickly.…”

Section: Discussionmentioning

confidence: 71%

Reliability and Validity of Smartphone Cognitive Testing for Frontotemporal Lobar Degeneration

Staffaroni,

Clark,

Taylor

et al. 2024

JAMA Netw Open

View full text Add to dashboard Cite

ImportanceFrontotemporal lobar degeneration (FTLD) is relatively rare, behavioral and motor symptoms increase travel burden, and standard neuropsychological tests are not sensitive to early-stage disease. Remote smartphone-based cognitive assessments could mitigate these barriers to trial recruitment and success, but no such tools are validated for FTLD.ObjectiveTo evaluate the reliability and validity of smartphone-based cognitive measures for remote FTLD evaluations.Design, Setting, and ParticipantsIn this cohort study conducted from January 10, 2019, to July 31, 2023, controls and participants with FTLD performed smartphone application (app)–based executive functioning tasks and an associative memory task 3 times over 2 weeks. Observational research participants were enrolled through 18 centers of a North American FTLD research consortium (ALLFTD) and were asked to complete the tests remotely using their own smartphones. Of 1163 eligible individuals (enrolled in parent studies), 360 were enrolled in the present study; 364 refused and 439 were excluded. Participants were divided into discovery (n = 258) and validation (n = 102) cohorts. Among 329 participants with data available on disease stage, 195 were asymptomatic or had preclinical FTLD (59.3%), 66 had prodromal FTLD (20.1%), and 68 had symptomatic FTLD (20.7%) with a range of clinical syndromes.ExposureParticipants completed standard in-clinic measures and remotely administered ALLFTD mobile app (app) smartphone tests.Main Outcomes and MeasuresInternal consistency, test-retest reliability, association of smartphone tests with criterion standard clinical measures, and diagnostic accuracy.ResultsIn the 360 participants (mean [SD] age, 54.0 [15.4] years; 209 [58.1%] women), smartphone tests showed moderate-to-excellent reliability (intraclass correlation coefficients, 0.77-0.95). Validity was supported by association of smartphones tests with disease severity (r range, 0.38-0.59), criterion-standard neuropsychological tests (r range, 0.40-0.66), and brain volume (standardized β range, 0.34-0.50). Smartphone tests accurately differentiated individuals with dementia from controls (area under the curve [AUC], 0.93 [95% CI, 0.90-0.96]) and were more sensitive to early symptoms (AUC, 0.82 [95% CI, 0.76-0.88]) than the Montreal Cognitive Assessment (AUC, 0.68 [95% CI, 0.59-0.78]) (z of comparison, −2.49 [95% CI, −0.19 to −0.02]; P = .01). Reliability and validity findings were highly similar in the discovery and validation cohorts. Preclinical participants who carried pathogenic variants performed significantly worse than noncarrier family controls on 3 app tasks (eg, 2-back β = −0.49 [95% CI, −0.72 to −0.25]; P &lt; .001) but not a composite of traditional neuropsychological measures (β = −0.14 [95% CI, −0.42 to 0.14]; P = .32).Conclusions and RelevanceThe findings of this cohort study suggest that smartphones could offer a feasible, reliable, valid, and scalable solution for remote evaluations of FTLD and may improve early detection. Smartphone assessments should be considered as a complementary approach to traditional in-person trial designs. Future research should validate these results in diverse populations and evaluate the utility of these tests for longitudinal monitoring.

show abstract

Section: Discussionmentioning

confidence: 71%

Reliability and Validity of Smartphone Cognitive Testing for Frontotemporal Lobar Degeneration

Staffaroni,

Clark,

Taylor

et al. 2024

JAMA Netw Open

View full text Add to dashboard Cite

show abstract

“…has been explored in previous research. Netzel and colleagues aimed to evaluate the test-retest reliability obtained when administering ImPACT to NCAA Division I athletes in a controlled laboratory versus an uncontrolled remote location [14] . Their ndings were consistent with the present study by concluding the ImPACT can be reliably administered to collegiate athletes in a remote environment.…”

Section: Discussionmentioning

confidence: 99%

Bringing ImPACT Home: Remote versus Traditional Administration for Collegiate Athletes

Bennett,

Slicer,

Stafford

et al. 2023

Preprint

View full text Add to dashboard Cite

Background When COVID-19 shut down our once-perceived “normal” societal function, university athletics were forced to change how they conducted concussion baseline screening for their student-athletes. The Immediate Post-Concussion Assessment and Cognitive Test (ImPACT) is a prominent assessment tool used amongst university athletic departments for identifying cognitive baseline functioning of collegiate athletes. The current study aimed to evaluate environmental and supervisory effects of neurocognitive baseline performance and overall symptom reports among student-athletes who took the ImPACT baseline assessment in a clinical setting or a remote environment. Results The four univariate ANCOVAs for Verbal Memory (p = 0.787), Visual Memory (p = 0.917), Visual Motor Speed (p = 0.261), and Impulse Control (p = 0.783) were not significantly different between groups. Additionally, the two Quade ANCOVAs for Reaction Time (p = 0.815) and Total Symptom Score (p = 0.886) were not significantly different. Conclusion Preliminary findings provide evidence and support that collegiate athletes can complete their baseline ImPACT in an unsupervised remote location without detriment to their performance.

show abstract

Minimal Detectable Change for the ImPACT Subtests at Baseline

Quigley,

Fenner,

Pavilionis

et al. 2024

Archives of Clinical Neuropsychology

View full text Add to dashboard Cite

Objective To establish the minimal detectable change (MDC) of the subtests that comprise the composite scores from remotely administered Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) baselines. Method Remote ImPACT baseline data from 172 (male = 45, female = 127) National Collegiate Athletic Association Division I student-athletes from the 2020 and 2021 athletic preseasons were used to calculate the MDC at the 95%, 90%, and 80% confidence intervals (CIs) for all subtest scores used to generate the four core composite scores and the impulse control composite. Results The MDCs for the verbal memory subtests at the 95% CI were 10.31 for word memory percent correct, 4.68 for symbol match total correct hidden, and 18.25 for three letters percentage correct. Visual memory subtest MDCs were 19.03 for design memory total percent correct and 4.90 for XO total correct memory. Visual motor speed subtest MDCs were 18.89 for XO total correct interference and 5.40 for three letters average counted correctly. Reaction time (RT) MDCs were 0.12 for XO average correct, 0.95 for symbol match average correct RT, and 0.28 for color match average correct. Impulse control MDCs were 5.97 for XO total incorrect and 1.15 for color match total commissions. One-way repeated measures MANOVA, repeated measures ANOVAs, and Wilcoxon signed-ranks test all suggested no significant difference between any subtests across two remote ImPACT baselines. Conclusions The ImPACT subtest scores did not significantly change between athletic seasons. Our study suggests the subtests be evaluated in conjunction with the composite scores to provide additional metrics for clinical interpretation.

show abstract

Test–Retest Reliability of Remote ImPACT Administration

Cited by 4 publications

References 36 publications

Reliability and Validity of Smartphone Cognitive Testing for Frontotemporal Lobar Degeneration

Reliability and Validity of Smartphone Cognitive Testing for Frontotemporal Lobar Degeneration

Bringing ImPACT Home: Remote versus Traditional Administration for Collegiate Athletes

Minimal Detectable Change for the ImPACT Subtests at Baseline

Contact Info

Product

Resources

About