ImPact Test-Retest Reliability: Reliably Unreliable?

Abstract

Context: Computerized neuropsychological testing is commonly used in the assessment and management of sport-related concussion. Even though computerized testing is widespread, psychometric evidence for test-retest reliability is somewhat limited. Additional evidence for test-retest reliability is needed to optimize clinical decision making after concussion.Objective: To document test-retest reliability for a commercially available computerized neuropsychological test battery (ImPACT) using 2 different clinically relevant time intervals.Design: Cross-sectional study. Setting: Two research laboratories.Patients or Other Participants: Group 1 (n ¼ 46) consisted of 25 men and 21 women (age ¼ 22.4 6 1.89 years). Group 2 (n ¼ 45) consisted of 17 men and 28 women (age ¼ 20.9 6 1.72 years).Intervention(s): Both groups completed ImPACT forms 1, 2, and 3, which were delivered sequentially either at 1-week intervals (group 1) or at baseline, day 45, and day 50 (group 2). Group 2 also completed the Green Word Memory Test (WMT) as a measure of effort.Main Outcome Measures: Intraclass correlation coefficients (ICCs) were calculated for the composite scores of ImPACT between time points. Repeated-measures analysis of variance was used to evaluate changes in ImPACT and WMT results over time.Results: The ICC values for group 1 ranged from 0.26 to 0.88 for the 4 ImPACT composite scores. The ICC values for group 2 ranged from 0.37 to 0.76. In group 1, ImPACT classified 37.0% and 46.0% of healthy participants as impaired at time points 2 and 3, respectively. In group 2, ImPACT classified 22.2% and 28.9% of healthy participants as impaired at time points 2 and 3, respectively.Conclusions: We found variable test-retest reliability for ImPACT metrics. Visual motor speed and reaction time demonstrated greater reliability than verbal and visual memory. Our current data support a multifaceted approach to concussion assessment using clinical examinations, symptom reports, cognitive testing, and balance assessment.Key Words: intraclass correlation, concussions, mild traumatic brain injuries, neuropsychological testing, athletes Key PointsImPACT had strong to weak test-retest reliability over time, consistent with the results of previous studies. Reliability was greater for the visual motor speed and reaction time subscores than for the verbal and visual memory subscores. Computerized neuropsychological testing is only 1 component of a multifaceted concussion-management program that uses all appropriate tools in clinical decision making.

False-Positive Rates of Reliable Change Indices for Concussion Test Batteries: A Monte Carlo Simulation

“…This is because indices with high correlations are unlikely to contribute uniquely to concussion assessment (and therefore to warrant separate inclusion in a battery) and because the difference scores that make up RCIs likely have lower maximum correlations with each other given that difference scores are often less reliable than single scores. 17 Authors 13,14 of the limited number of published studies on this topic have reported that 22.2% to 46% of healthy participants produce at least 1 (out of 5) significantly declined RCIs on ImPACT.…”

If You're Not Measuring, You're Guessing: The Advent of Objective Concussion Assessments

“…46 Injury sensitivity improved from 23% to 44% 10,11 on pencil-and-paper tests to 63% to 82% 10,47 on the cognitive components of computer-based tests, but concerns about reliability and false-positive findings hampered recommendations for their isolated use. [48][49][50][51] Ultimately, given the time pressures of sports medicine, practitioners were quick to adopt computer-based testing. Indeed, in 2001, only 15% of ATs in all settings were using any form of neurocognitive testing in the management of patients with concussive injuries.…”

Reliability of Computerized Neurocognitive Tests for Concussion Assessment: A Meta-Analysis

“…18 The controversy of CNT reliability is further complicated because of the wide range of reliability estimates (from as low as 0.10 to as high as 0.93) reported across studies. 7,8,11,12,[17][18][19][20][21][22][23][24][25][26][27][28][29][30] Because of the conflicting reports, a more thorough investigation of the reliability of CNT scores is necessary. Although a systematic review 10 summarizing the reliability of ImPACT scores has been previously published, this study was limited to a single CNT.…”