Performance on the Cognitive Reflection Test is stable across time

Abstract

A widely used measure of individual propensity to utilize analytic processing is the Cognitive Reflection Test (CRT), a set of math problems with intuitively compelling but incorrect answers. Here, we ask whether scores on this measure are temporally stable. We aggregate data from 11 studies run on Amazon Mechanical Turk in which the Cognitive Reflection Test (CRT) was administered and identify N = 3,302 unique individuals who completed the CRT two or more times. We find a strong correlation between an individual’s first and last CRT performance, r = .806. This remains true even when constraining to data points separated by over 2 years, r = .755. Furthermore, we find that CRT scores from one timepoint correlated negatively with belief in God and social conservatism from the other timepoint (and to a similar extent as scores gathered at the same timepoint). These results show that CRT scores are stable over time, and – given the stable relationship between CRT and religious belief and ideology – provide some evidence for the stability of analytic cognitive style more generally.

Reflection increases belief in God through self-questioning among non-believers

“…The results also showed that those who make greater use of their reflective capacities (as measured by CRT-2) are less likely to endorse belief in God or gods. These results provide evidence against the hypothesis that intuition fosters and that reflection dampens religious belief (Gervais & Norenzayan, 2012;Shenhav et al, 2012; but it converges with the longstanding correlational results demonstrating that tendency for reflective thinking is negatively associated with religious belief (e.g., Bahçekapili & Yilmaz, 2017;Gervais et al, 2018;Pennycook et al, 2016;Stagnaro et al, 2018;Stagnaro, Ross, Pennycook & Rand, 2019).…”

“…Correlational findings often show a negative relationship between religious belief and reflective (i.e., analytic) thinking style (Bahçekapili & Yilmaz, 2017;Gervais & Norenzayan, 2012;Gervais et al, 2018;Pennycook, Cheyne, Seli, Koehler & Fugelsang, 2012;Stagnaro, Pennycook & Rand, 2018;Yilmaz & Saribay, 2016). Similarly, a recent meta-analysis found that the analytic thinking performance of non-believers (atheists and agnostics) was on average higher than that of religious believers (Pennycook, Ross, Koehler & Fugelsang, 2016).…”

The link between intuitive thinking and social conservatism is stronger in WEIRD societies

“…There are intuitive but incorrect answers to these questions. If one can suppress the intuitive answer and give the correct answer, this is considered an indicator of analytic thinking (e.g., Stagnaro, Pennycook & Rand, 2018;Meyer, Zhou & Frederick, 2018). Many Labs 2 Project included three CRT questions (Finucane & Gullion, 2010; see Table 1).…”

“…Another limitation of this research is that it uses only three numerical CRT questions to measure ACS. Although there are some recent findings indicating that CRT corresponds to a stable personality trait (Meyer et al, 2018;Stagnaro et al, 2018), there are also some others claiming that it sometimes does not measure relevant aspects of ACS (e.g., Baron, Scott, Fincher & Metz, 2015). Although the results generally remained constant when other ACS measures were used (e.g., Pennycook et al, 2014), Yilmaz and Saribay (2017c) found that the three different ACS measures (CRT-2, Base rate Conflict Problems, and Actively Open Minded-Thinking) were significantly (and negatively) related to social conservatism, and that CRT was not individually related to any of the ideology measures (including social conservatism).…”

Turning up the heat: The impact of indoor temperature on selected cognitive processes and the validity of self-report

“…Finally, the choice for our test battery is the outcome of a careful trade-off between practical and theoretical considerations. Research has suggested that the CRT is robust under multiple exposure (Bialek & Pennycook, 2017;Meyer, Zhou & Frederick, 2018) and consistent over time (Stagnaro et al, 2018). Recognition of the original CRT is relatively high (46% recognized at least one question, and 20% recognized all questions) .…”