Does the cognitive reflection test measure cognitive reflection? A mathematical modeling approach

Abstract

We used a mathematical modelling approach, based on a sample of 2,019 participants, to better understand what the cognitive reflection test (CRT, Frederick, 2005) measures. This test, which is typically completed in less than ten minutes, contains three problems, and aims to measure the ability or disposition to resist reporting the response that first comes to mind.However, since the test contains three mathematical based problems, it is possible that the test only measures mathematical abilities and not cognitive reflection. We found that the models that included an inhibition parameter (i.e., the probability of inhibiting an intuitive response), as well as a mathematical parameter (i.e., the probability of using an adequate mathematical procedure), fitted the data better than a model that only included a mathematical parameter. We also found that the inhibition parameter in males is best explained by both rational thinking ability and the disposition towards actively open-minded thinking, whereas in females this parameter was better explained by rational thinking only.With these findings this study contributes to the understanding of the processes involved in solving CRT, and will be particularly useful for researchers who are considering using this test in their research. Frederick (2005) with the purpose of measuring the construct cognitive reflection, which he defined as "the ability or disposition to resist reporting the response that first comes to mind" (Frederick, 2005, p. 35). KeywordsAs shown in Table 1 CRT contains three mathematical problems with the common feature that they all typically trigger a quick, intuitive response, which is not the correct answer. If the test taker realises that the intuitive response is not the correct answer, finding the correct solution requires relatively easy mathematical computations. Typically, a participant either solves a problem incorrectly or correctly within a few minutes. Research has shown that people find it difficult to solve these problems, and that those who perform well at CRT tend to perform well at numeracy tests, other general ability tests, and tend to avoid biases in Frederick's definition of cognitive reflection is intriguing because it encompasses the possibility that cognitive reflection is a thinking disposition. As noted by Toplak et al. (2011) thinking dispositions are typically measured with subjective reports, which are not always reliable (e.g., Nisbett & Wilson, 1977). CRT is a performance measure with an objective criterion. Thus, if the CRT, indeed, measures a thinking disposition it would constitute a substantial progress in measuring thinking dispositions.Researchers seem to disagree in whether CRT measures an ability or both an ability and a disposition. Cokely and Kelley (2009) Another group of researchers seem to view CRT as a measure of an ability (not a disposition), but they consider this ability as distinct from general cognitive abilities (e.g., intelligence, working memory). Toplak et al. (2011) referred to this ab...

Are there gender differences in cognitive reflection? Invariance and differences related to mathematics

“…The results confirmed that, in line with several previous studies (e.g., Campitelli & Gerrans, 2014;Cueva et al, 2016;Frederick, 2005;Pennycook et al, 2016;Primi et al, 2016), gender was related to performance on the CRT (i.e., men scored significantly higher than women). However, the gender differences disappeared when numerical skills and math-related anxiety were statistically controlled.…”

“…However, the gender differences disappeared when numerical skills and math-related anxiety were statistically controlled. Although the CRT is more than just a test of numeracy (e.g., Campitelli & Gerrans, 2014;Liberali et al, 2012), these results suggest that the numerical content of the problems acts as a confounding variable .…”

“…Additionally, previous studies (see Su arez-Pellicioni, N uñez-Peña, & Colom e, 2016, for a review) have shown attentional control problems in high math-anxious individuals, especially in inhibitory tasks (Stroop tasks), which could also explain the tendency to rely on salient heuristics. Indeed, several authors have noted that inhibition of the easily available heuristic response is necessary for correct performance on the CRT (e.g., B€ ockenholt, 2012;Campitelli & Gerrans, 2014;Travers et al, 2016). Morsanyi et al (2014) also explored the possibility that MA affected performance on the CRT through burdening working memory resources.…”

The cognitive reflection test is robust to multiple exposures

“…The correct response is 5 cents, an answer that is reached by only roughly 30% of university undergraduates, depending on the university (De Neys, Rossi, & Houdé, 2013;Frederick, 2005;Pennycook, Cheyne, Koehler, & Fugelsang, 2016). However, errors are also not random: Almost all who get CRT questions wrong give the intuitive response (Campitelli & Gerrans, 2014;Frederick, 2005;Pennycook, Cheyne, et al, 2016). Moreover, the majority of participants who answer correctly are aware of the incorrect intuitive answer, whereas those who got it wrong naturally failed to consider the correct answer (Mata, Ferreira, & Sherman, 2013;Pennycook, Ross, Koehler, & Fugelsang, 2017).…”

Is the cognitive reflection test a measure of both reflection and intuition?

“…Nonetheless, there is strong evidence that the CRT is not just another numeracy test (Campitelli & Gerrans, 2014;Cokely & Kelley, 2009;Liberali, Reyna, Furlan, Stein, & Pardo, 2011;Toplak, West, & Stanovich, 2011; but see Weller et al, 2013). Under the assumption that one must engage reflective reasoning processes to override a prepotent intuitive response, the willingness or disposition to engage Type 2 processing should be an important determinant of performance.…”