The cognitive reflection test is robust to multiple exposures

Abstract

The cognitive reflection test (CRT) is a widely used measure of the propensity to engage in analytic or deliberative reasoning in lieu of gut feelings or intuitions. CRT problems are unique because they reliably cue intuitive but incorrect responses and, therefore, appear simple among those who do poorly. By virtue of being composed of so-called Btrick problems^that, in theory, could be discovered as such, it is commonly held that the predictive validity of the CRT is undermined by prior experience with the task. Indeed, recent studies have shown that people who have had previous experience with the CRT score higher on the test. Naturally, however, it is not obvious that this actually undermines the predictive validity of the test. Across six studies with~2,500 participants and 17 variables of interest (e.g., religious belief, bullshit receptivity, smartphone usage, susceptibility to heuristics and biases, and numeracy), we did not find a single case in which the predictive power of the CRT was significantly undermined by repeated exposure. This occurred despite the fact that we replicated the previously reported increase in accuracy among individuals who reported previous experience with the CRT. We speculate that the CRT remains robust after multiple exposures because less reflective (more intuitive) individuals fail to realize that being presented with apparently easy problems more than once confers information about the task's actual difficulty.

Measuring Cognitive Reflection without Maths: Development and Validation of the Verbal Cognitive Reflection Test

“…First, even though our items have low initial familiarity, with the continuous use of the measure, familiarity will become an issue. Our research did not establish whether familiarity will be detrimental to the predictive validity of the Verbal CRT or whether it will only have a negligible effect on predictive validity as has been demonstrated recently with the Numerical CRT (Bialek & Pennycook, 2017;Meyer et al, 2018). Future research should address the issue of previous exposure and its effect on predictive validity.…”

“…Second, an increasing proportion of participants are already familiar with the CRTin terms of prior exposure as well as in terms of knowledge of the items. Self-reported prior exposure substantially increased the average performance in the test (e.g., Bialek & Pennycook, 2017;Haigh, 2016;Stieger & Reips, 2016). However, recent studies have reported that self-reported prior exposure did not diminish the predictive validity of the test (Bialek & Pennycook, 2017) and found no evidence that the actual prior exposure (not self-reported) increased the performance (Meyer, Zhou, & Frederick, 2018).…”

Assessing the validity of inferences from scores on the cognitive reflection test

“…One limitation is that research pools may be polluted due to overexposure to the CRT items (Haigh, 2016). However, recent studies (i.e., Bialek & Pennycook, 2017) found that exposure does not negatively affect results. Further, we used a seven-item measure to address this issue as Haigh (2016) suggested.…”

“…The CRT is widely used because it was uniquely designed to measure the intuitive thinking processes linked to cognitive biases (Frederick, 2005) and has been shown to predict variance beyond general mental ability (g) in decision-making outcomes (Toplak, West, & Stanovich, 2014). In addition, as concern over the influence of cognitive bias on decision making is growing (e.g., Comaford, 2016;National Research Council, 2015), the CRT has received increased attention as a meaningful predictor of susceptibility to cognitive heuristics in both the scientific community (e.g., Bialek & Pennycook, 2017) and the general public (e.g., Cowen, March 2018).…”

Unhealthy and unaware? Misjudging social comparative standing for health-relevant behavior

“…We also added the 7-item expanded version of the Cognitive Reflection Test (CRT; Toplak, West, & Stanovich, 2014) in Study 2 to examine if individual differences in cognitive capability could be related to percentile miscalibration. We chose the CRT because it has shown to be significantly associated with numeracy (Bialek & Pennycook, 2018;Liberali, Reyna, Furlan, Stein, & Pardo, 2012) and metacognitive monitoring (Mata, Ferreira, & Sherman, 2013;Pennycook, Ross, Koehler, & Fugelsang, 2017). Performance on the CRT can also be used as a metric of general mental ability (Blacksmith, Yang, Behrend, & Ruark, 2019), as well as a marker of people's willingness to think analytically and use their cognitive resources (Frederick, 2005;Šrol, 2018).…”