Test-Retest Reliability of Four Computerized Neurocognitive Assessment Tools in an Active Duty Military Population

Abstract

Computerized neurocognitive assessment tools (NCATs) are increasingly used for baseline and post-concussion assessments. To date, NCATs have not demonstrated strong test-retest reliabilities. Most studies have used non-military populations and different methodologies, complicating the determination of the utility of NCATs in military populations. The test-retest reliability of four NCATs (Automated Neuropsychological Assessment Metrics 4 [ANAM4], CNS-Vital Signs, CogState, and Immediate Post-Concussion Assessment and Cognitive Test [ImPACT]) was investigated in a healthy active duty military sample. Four hundred and nineteen Service Members were randomly assigned to take one NCAT and 215 returned after approximately 30 days for retest. Participants deemed to have inadequate effort during one or both testing sessions, according to the NCATs scoring algorithms, were removed from analyses. Each NCAT had at least one reliability score (intraclass correlation) in the "adequate" range (.70-.79), only ImPACT had one score considered "high" (.80-.89), and no scores met "very high" criteria (.90-.99). However, overall test-retest reliabilities in four NCATs in a military sample are consistent with reliabilities reported in the literature and are lower than desired for clinical decision-making.

Reliability of Computerized Neurocognitive Tests for Concussion Assessment: A Meta-Analysis

“…Furthermore, the proportion of outcomes with acceptable reliability was calculated for each CNT. Only 2 studies 21,40 examined the reliability of the CNS-VS test, whereas only a single study 11 examined the reliability of Headminder. The samples for these CNTs were not included in the overall meta-analysis because of the small sizes.…”

“…To increase power and account for a small sample size, a within-subject study design is often used to compare CNTs. This practice of examining multiple CNTs in a single study 11,14,21 has been questioned by some due to the high risk of fatigue from extended test protocols.…”

“…These findings are in contrast to those of Schatz et al, 14 who proposed that the low reliability in some studies is related to cognitive fatigue and low methodologic scrutiny. It is likely that the differences in ES estimates identified by the studies in question 11,21 are related to differences in analytic methods rather than to differences in study design. Many of the studies included in this meta-analysis used different ICC models for analyzing test-retest reliability.…”

“…18 The controversy of CNT reliability is further complicated because of the wide range of reliability estimates (from as low as 0.10 to as high as 0.93) reported across studies. 7,8,11,12,[17][18][19][20][21][22][23][24][25][26][27][28][29][30] Because of the conflicting reports, a more thorough investigation of the reliability of CNT scores is necessary. Although a systematic review 10 summarizing the reliability of ImPACT scores has been previously published, this study was limited to a single CNT.…”

Blast Injury

“…56 Several questionnaire-based screening tools have been developed to identify mTBI. 53,57,58 Identification of casualties with even mTBI is important because of the benefits of cognitive rehabilitation and the need to provide support for what may be lifelong, if subtle, neurocognitive problems. 59,60 Rehabilitation and Outcomes Individuals with significant injuries related to blast trauma should be evaluated for rehabilitation after their acute injuries are stabilized.…”