False-Positive Rates of Reliable Change Indices for Concussion Test Batteries: A Monte Carlo Simulation

Abstract

Background: Neurocognitive testing is widely performed for the assessment of concussion. Athletic trainers can use preseason baselines with reliable change indices (RCIs) to ascertain whether concussed athletes' cognitive abilities are below preinjury levels. Although the percentage of healthy individuals who show decline on any individual test is determined by its RCI's confidence level (eg, 10% false-positive rate using an RCI with an 80% confidence interval), the expected rate of 1 or more significant RCIs across multiple indices is unclear.Objective: To use a Monte Carlo simulation procedure to estimate the normal rate (ie, base rate) of significant decline on 1 or more RCIs in multitest batteries.

Prospective, Head-to-Head Study of Three Computerized Neurocognitive Assessment Tools (CNTs): Reliability and Validity for the Assessment of Sport-Related Concussion

“…First, the clinical manifestations of SRC are most prominent immediately after injury and demonstrate rapid recovery even within the first hours post-injury at a group level (McCrea et al, 2003). Indeed, our findings are consistent with prior meta-analyses of the magnitude of neurocognitive changes after SRC (Belanger & Vanderploeg, 2005; Broglio & Puetz, 2008) and with what is known about the rapid clinical recovery course after concussion (for a review, see Nelson, Janecek, & McCrea, 2013). An alternative viewpoint is that impairments persist further out from injury but that these CNTs simply lack the sensitivity to detect the abnormal signal.…”

“… 3 As has been thoroughly examined in another report on the larger baseline sample from this study (Nelson et al, 2015), MSVT failure was rare and demonstrated poor agreement with the validity output of any CNT. Thus, given our goal to examine the performance of these CNTs in their typical clinical context (in which only the CNT validity criteria are available, and because the MSVT does not appear to measure the same construct as related to performance validity as the CNTs measure, and, we did not exclude subjects from the primary analyses due to failure to pass the MSVT at baseline ( n =3 in this sample).…”

“…However, given that clinicians using these multi-index batteries are faced with interpreting the results of sets of indices simultaneously, it is critical to know the joint base rates of impairment in healthy controls (i.e., false positives) to fairly judge the utility of the tests and to identify optimal decision rules for classifying individuals as impaired. Although the false positive rates of individual RCIs can be predicted from their confidence levels (e.g., 10% using an 80% CI; 5% using a 90% CI), as with any set of neuropsychological tests, the base rates of impairment across multiple tests may be much higher depending on the number of indices being jointly interpreted and their intercorrelations (Crawford, Garthwaite, & Gault, 2007; Nelson, in press; Schretlen, Testa, Winicki, Pearlson, & Gordon, 2008).…”

“…Participants were contact and collision sport athletes from 9 high schools and 4 colleges in southeastern Wisconsin enrolled in Project Head to Head between August, 2012 and October, 2014 (see also LaRoche, Nelson, Connelly, Walter, & McCrea, 2015; Nelson, Pfaller, Rein, & McCrea, 2015). Among the 2,148 participants who consented to participate, 166 were concussed during the study and were enrolled in post-injury testing.…”

Cognitive Outcome 1 Year After Mild Traumatic Brain Injury

“…It is important to note that our concise cognitive battery is less susceptible to false-positive classification as "cognitive decline" with the RCI method because the false-positive rate increases with longer test batteries. 22 Definition of 1-Year Poor Cognitive Outcome Our primary definition of poor 1-year cognitive outcome was cognitive impairment (≥1 score on ≥2 of 3 tests meeting criteria for impairment), cognitive decline (≥1 score on ≥2 of 3 tests meeting criteria for decline), or both. This cognitive outcome categorization for each patient with mTBI and control participant was visualized by first stratifying the cohort by 1-year cognitive impairment status and then plotting the RCI used to determine cognitive decline categorization for each patient (i.e., RCI from the test with the second greatest decline).…”