Most evidence for the compensation account of cognitive training is unreliable

Abstract

Cognitive training and brain stimulation studies have suggested that human cognition, primarily working memory and attention control processes, can be enhanced. Some authors claim that gains (i.e., post-test minus pretest scores) from such interventions are unevenly distributed among people. The magnification account (expressed by the evangelical "who has will more be given") predicts that the largest gains will be shown by the most cognitively efficient people, who will also be most effective in exploiting interventions. In contrast, the compensation account ("who has will less be given") predicts that such people already perform at ceiling, so interventions will yield the largest gains in the least cognitively efficient people. Evidence for this latter account comes from reported negative correlations between the pretest and the training/stimulation gain. In this paper, with the use of mathematical derivations and simulation methods, we show that such correlations are pure statistical artifacts caused by the widely known methodological error called "regression to the mean". Unfortunately, more advanced methods, such as alternative measures, linear models, and control groups do not guarantee correct assessment of the compensation effect either. The only correct method is to use direct modeling of correlations between latent true measures and gain. As to date no training/stimulation study has correctly used this method to provide evidence in favor of the compensation account, we must conclude that most (if not all) of the evidence should be considered inconclusive.

A Systematic Review on Predictors of Working Memory Training Responsiveness in Healthy Older Adults: Methodological Challenges and Future Directions

“…However, no causal interferences can be derived from correlation analyses (Bewick et al, 2003 ). Furthermore, correlations, for example, between baseline performance and change scores (which is obtained by subtracting baseline performance from post-training performance), are less more than pure statistical artifacts (Smoleń et al, 2018 ). Smoleń et al ( 2018 ) discuss that, unfortunately, even more advanced methods such as multiple regressions and linear mixed models do not guarantee the correct assessment of relationships between predictor variables and respective outcomes.…”

“…Furthermore, correlations, for example, between baseline performance and change scores (which is obtained by subtracting baseline performance from post-training performance), are less more than pure statistical artifacts (Smoleń et al, 2018 ). Smoleń et al ( 2018 ) discuss that, unfortunately, even more advanced methods such as multiple regressions and linear mixed models do not guarantee the correct assessment of relationships between predictor variables and respective outcomes. According to the authors, the only correct method would be to use direct modeling of correlations between latent true measures and gain by structural equation modeling (Smoleń et al, 2018 ).…”

“…Smoleń et al ( 2018 ) discuss that, unfortunately, even more advanced methods such as multiple regressions and linear mixed models do not guarantee the correct assessment of relationships between predictor variables and respective outcomes. According to the authors, the only correct method would be to use direct modeling of correlations between latent true measures and gain by structural equation modeling (Smoleń et al, 2018 ). Future research on prognostic factors regarding (working memory) training responsiveness should apply advanced statistical methods such as latent difference score models or growth curve analyses as highly flexible statistical approaches from the structural equation modeling background.…”

Nothing wrong about change: The adequate choice of the dependent variable and design in prediction of cognitive training success

“…According to these contradictory results, two explanatory accounts have occurred: the magnification account, which predicts that cognitively efficient people also show the most gain in nonpharmacological interventions; and the compensation account, which states that interventions will yield the largest gain in the least cognitively efficient people [16]. However, a current paper of Smolén et al (2018) could show, that most evidence for the compensation account of nonpharmacological training interventions is unreliable due to methodological errors in the original studies [17]. Also, a recent systematic review on prognostic factors of memory training success in healthy older adults could show that the results vary not only as a function of the type of statistical calculation used to determine prognostic factors, but also of the type of dependent variables used in the calculations [18]: post-test scores, change scores, residual change scores, and relative change scores.…”

“…Smoleń et al (2018) notes that many of the correlations between pre-test scores and absolute change scores reported in the literature to support the compensation account are suspiciously high, especially considering the theoretical limit of observable correlations given the imperfect reliability of psychological measures[17]. Here, we have demonstrated that these high correlations might in fact reflect low reliabilities of the measures used in the respective studies, which is in line with Smoleń's mathematical demonstrations of why negative correlations between pre-test scores and absolute change scores emerge naturally[17].DiscussionAs prognostic research and especially studies on the impact of parameters predicting the success of CT (or in general pharmacological and nonpharmacological interventions) have become of huge scientific interest over the past few years, the present paper aimed at systematically showing and discussing different types of regression models and dependent variables used, as well as the influence of reliability of measures, sample sizes, and the specific role of baseline measurements (pre-test scores) as predictors in multiple regressions to account for changes after interventions. With the help of simulation methods and mathematical derivations we could show that (Aim 1) a regression model including P-I, Group, pre-test score, and P-I × Group as predictors seems most convenient when investigating predictors of changes interventions such as CT, as well as (Aim 2) using the absolute change scores as the dependent variable.…”