A reflection on cognitive reflection – testing convergent validity of two versions of the Cognitive Reflection Test

Abstract

Although it is generally acknowledged that the Cognitive Reflection Test (CRT) captures intelligence and numerical ability, many agree that it cannot be completely reduced to these constructs. Rather, it is presumed that the CRT also assesses some kind of thinking disposition towards reflective and open-minded thinking. In this manuscript, we report the results of a study that tested this assumption by exploring convergent validity of both the numerical and verbal version of the CRT. Using structural equation modelling, we investigated whether intelligence and numerical ability can account for all the variance in the CRT and if not, what is the nature of the unaccounted variance. Our conclusions about the convergent validity differed for the two types of test. For the numerical CRT, we found that the correlation between the latent numerical CRT and numerical ability was so high that the constructs were practically indistinguishable. As for the verbal CRT, the correlations between the latent verbal CRT and intelligence and numerical ability constructs were substantially lower, meaning that these two constructs do not account for all the variance in the test. However, the latent verbal CRT failed to correlate with belief bias and actively open-minded thinking, two closely related constructs, once the variance of intelligence and numerical ability was partialled out. We concluded that, despite its name, the CRT does not seem to assess the construct of cognitive reflection and its correlation with other variables found in the literature might mostly be driven by its overlap with intelligence and numerical ability.

A diffusion model analysis of belief bias: Different cognitive mechanisms explain how cognitive abilities and thinking styles contribute to conflict resolution in reasoning

“…However, we also found that CRT performance was more strongly related to intelligence test performance, r = .33, than to need for cognition, r = -.18. This finding is not a fluke in our data, but rather it is consistent with a recent study that systematically evaluated the validity of two CRT versions and concluded that they did not access the construct of cognitive reflection, but rather individual differences in intelligence and numerical abilities (Erceg et al, 2020).…”

“…In addition to drift rate v and boundary separation a, we also estimated the non-decision time t0 and the trial-to-trial variability of the non-decision time st0. The starting point z was fixed to a/2 and all other model parameters 1 The current debate regarding the CRT reflects uncertainty regarding its validity: While some argue that the CRT measures individual differences in reflective thinking ability (Frederick, 2005), others argue that it largely measures cognitive and numerical abilities (Blacksmith et al, 2019;Erceg et al, 2020). Undoubtedly, however, performance in the CRT predicts susceptibility to biases and logical fallacies (e.g., Bialek & Pennycook, 2018;Pennycook & Ross, 2016), making it an important measure to consider in any study on individual differences in bias susceptibility.…”

“…Hence, we could not test whether individual differences in cognitive reflection and need for cognition jointly or independently moderated conflict resolution. Because successfully solving CRT items requires both algorithmic and reflective thinking (Erceg et al, 2020;Stanovich, 2009Stanovich, , 2010Toplak et al, 2011), it is possible that CRT solution rates only predicted successful conflict resolution to the degree the it measured reflective thinking ability. However, to test this interpretation of our results, a larger sample size suited for multiple regression analysis would be needed.…”

“…Moreover, Erceg et al (2020) also failed to find any association between CRT scores and reasoning performance in a belief bias task once individual differences in intelligence and numerical abilities were accounted for. In addition, recent findings have indicated that the CRT may only measure individual differences in cognitive reflection in less intelligent individuals who do not possess strong logical intuitions, but that it is not an accurate measure of cognitive reflection in more intelligent individuals with strong numerical abilities (Erceg et al, 2019).…”

Your Health vs. My Liberty: Philosophical beliefs dominated reflection and identifiable victim effects when predicting public health recommendation compliance during the COVID-19 pandemic

“…Univariate analysis also found common correlations. Reflection test performance correlated with numeracy (Attali & Bar-Hillel, 2020;Białek, Bergelt, et al, 2019;Białek & Sawicki, 2018;Erceg et al, 2020;Patel, 2017;Szaszi et al, 2017), morality Patil et al, 2018;Reynolds et al, 2019), political orientation (Deppe et al, 2015;Yilmaz et al, 2019), and theism (Gervais et al, 2018;Pennycook et al, 2016). Further, older people were more conservative (Cornelis et al, 2009;Luberti et al, 2020) and women were less likely to endorse instrumental harm (e.g., Friesdorf et al, 2015).…”