Scoring Higher the Second Time Around: Meta-Analyses of Practice Effects in Neuropsychological Assessment

Abstract

In neuropsychological assessment, and many areas of research, it is common for the same test to be administered on more than one occasion to measure change. Measured changes are presumed to reflect true changes in the construct being measured by the test; for example, cognitive changes due to processes such as aging, advancing neurological disease, or treatment interventions. However, practice effects, defined as score increases due to factors such as memory for specific test items, learned strategies, or test sophistication, complicate the interpretation of change. This review presents meta-analyses of nearly 1600 individual effect sizes representing changes in mean-level performance on tests commonly used to assess core domains of neuropsychological function, with the goal of quantitatively summarizing the magnitude of practice effects on such tests. The use of alternate forms, the ages of participants, clinical diagnoses of study participants, and length of the test-retest interval were associated with the magnitude of change in many cases. These findings have important implications for the practice of clinical neuropsychology, as well as for research applications, and highlight the need for practice effects to be taken into account in interpreting change across time with multiple measurements.

ECAS A-B-C: alternate forms of the Edinburgh Cognitive and Behavioural ALS Screen

“…This finding is in agreement with recent research demonstrating that the ECAS-A is susceptible to practice effects with repeated administration (32). The present study was designed to maximize the possible detection of practice effects as short intervals have been shown to exacerbate such an effect (29). However, no significant change in performance was detected over time when alternate versions of the ECAS were administered.…”

“…As practice effects have shown susceptibility to short retest intervals (e.g. see Calamia, Markon, & Tranel, 2012) and to maximize the possibility of detecting such effects, participants were administered the ECAS repeatedly during the same sitting. Phase 3 testing for each participant lasted approximately 50 min, limiting the possibility of fatigue.…”

“…As such, the accurate assessment of cognition and behaviour over time is of vital importance to meeting the needs of patients and their families. However, it has been well documented that the repeated administration of the same neuropsychological test can result in an improvement in performance (29). This improvement, termed practice effects, may result from 1) learning the content of test items, e.g., remembering the content of a prose story to be remembered; and 2) development of test-taking strategies (30,31).…”

Establishing normative data for repeated cognitive assessment: A comparison of different statistical methods

“…Ideally, the observed changes in the test scores at subsequent measurement occasions would be directly interpretable in terms of true changes in the latent cognitive trait of interest. This is, however, generally not the case (Calamia, Markon, & Tranel, 2012). The main reason for this is that practice effects occur in serial testing situations.…”

“…The main reason for this is that practice effects occur in serial testing situations. Practice effects refer to a variety of factors-such as procedural learning, memory for specific items, and increased comfort with formal testing situations (McCaffrey, Duff, & Westervelt, 2000)that result in systematic improvements in test scores at retesting occasions, even though there was no true change in the latent trait that is measured by the cognitive test (Bartels, Wegrzyn, Wiedl, Ackermann, & Ehrenreich, 2010;Calamia et al, 2012;Van der Elst, Van Breukelen, Van Boxtel, & Jolles, 2008). Practice effects are especially pronounced when the testretest intervals are short (e.g., Theisen, Rapport, Axelrod, & Brines, 1998), but they also occur in studies with test-retest intervals of several years (Rönnlund & Nilsson, 2006;Salthouse, Schroeder, & Ferrer, 2004).…”