Measuring Cognitive Reflection without Maths: Development and Validation of the Verbal Cognitive Reflection Test

Abstract

<p>The Cognitive Reflection Test (CRT) became popular for its impressive power to predict how well people reason and make decisions. Despite the popularity of the CRT, a major issue complicates its interpretation: the numerical nature of the CRT confounds reflection ability with mathematical ability. We have addressed this issue by developing the Verbal CRT (CRT-V), a novel 10-item measure of cognitive reflection, using non-mathematical problems with good statistical and psychometric properties and with low familiarity. First, we selected suitable items with relatively low familiarity and optimal difficulty as identified in two different populations (Studies 1 and 2) and with high content validity as judged by an expert panel (Study 3). Second, we demonstrated a good criterion and construct validity for the test in different populations with a wide range of variables (Studies 4-6, 8) and a good internal consistency and test-retest reliability (Study 7). The Verbal CRT was less associated with numeracy than the original CRT and it was test equivalent across gender, age groups and administration setting. In contrast with the original CRT (Hedge’s g = 0.29, 95% CI[0.17, 0.40]), the Verbal CRT showed no gender differences (Hedge’s g = -0.06, 95% CI[-0.18, 0.06]). The Verbal CRT can complement existing tests of cognitive reflection.</p>

Who detects and why: How individual differences in cognitive characteristics underpin different types of responses on reasoning tasks?

“…Perhaps then we should turn to the problems that are similar to the CRT problems in that they cue incorrect intuitive response, but also do not allow that much opportunity to develop expertise and perhaps draw from different abilities than numeracy and math knowledge. Belief bias syllogisms, verbal CRT (Sirota, Kostovičová, Juanchich, Dewberry, & Marshall, 2018) or some heuristics and biases tasks could be a way to go. However, it is a question whether cuing an initial wrong response is even an important feature if the task is to measure reflection as perhaps people would generally benefit from slowing down and reflecting on one's answers on any type of task.…”

The Relationship Between an Alternative Form of Cognitive Reflection Test and Intertemporal Choice

“…Recently, a new version of CRT (termed verbal CRT) based on non-mathematical problems was developed. This version has a weaker relationship with numeracy and is more gender neutral (Sirota, Kostovičová, Juanchich, Dewberry, & Marshall, 2018). We believe developing such a version is the right step to tease apart cognitive reflection and numeracy.…”

Mental Simulation as a Remedy for Biased Reasoning

“…The objective of the other pilot study (N = 26) was to prepare two short cognitive reflection tests (CRT) of roughly the same difficulty. We tested the original CRT tasks (Frederick, 2005), their alternatives with a modified wording (Kostovičová, Dudeková, & Konečný, 2013), new items from the extended version (Toplak et al, 2014), and verbal CRT problems (Sirota et al, 2018). We checked the items for their difficulty, discrimination power, content clarity and familiarity.…”

“…Based on the results, we designed two versions, consisting of 2 verbal and 2 numerical tasks, for instance: "How many of each animal did Moses take on the ark?" (Sirota et al, 2018) or "Jerry received both the 15th highest and the 15th lowest mark in the class. How many students are in the class?"…”

“…As a proxy for intuitive versus analytic thinking we chose cognitive reflection. A high score on the cognitive reflection tests (Frederick, 2005;Sirota, Kostovičová, Juanchich, Dewberry, & Marshall, 2018;Toplak, West, & Stanovich, 2014) means that the person is able to exert cognitive effort, suppress their automatic incorrect responses, and engage in deliberative thinking. It is a very strong predictor of susceptibility to cognitive biases (e.g., Toplak, West, & Stanovich, 2011, rational choices and reallife decision outcomes (e.g., Campitelli & Labollita, 2010;Juanchich, Dewberry, Sirota, & Narendran, 2016).…”