Measuring Probabilistic Reasoning: The Construction of a New Scale Applying Item Response Theory

Abstract

Probabilistic reasoning skills are important in various contexts. The aim of the present study was to develop a new instrument (the Probabilistic Reasoning Scale – PRS) to accurately measure low levels of probabilistic reasoning ability in order to identify people with difficulties in this domain. Item response theory was applied to construct the scale, and to investigate differential item functioning (i.e., whether the items were invariant) across genders, educational levels, and languages. Additionally, we tested the validity of the scale by investigating the relationships between the PRS and several other measures. The results revealed that the items had a low level of difficulty. Nonetheless, the discriminative measures showed that the items can discriminate between individuals with different trait levels, and the test information function showed that the scale accurately assesses low levels of probabilistic reasoning ability. Additionally, through investigating differential item functioning, the measurement equivalence of the scale at the item level was confirmed for gender, educational status, and language (i.e., Italian and English). Concerning validity, the results showed the expected correlations with numerical skills, math‐related attitudes, statistics achievement, IQ, reasoning skills, and risky choices both in the Italian and British samples. In conclusion, the PRS is an ideal instrument for identifying individuals who struggle with basic probabilistic reasoning, and who could be targeted by specific interventions. Copyright © 2017 John Wiley & Sons, Ltd.

The link between deductive reasoning and mathematics

“…The two tasks that assessed mathematical reasoning about word problems were the Cognitive Reflection Test-Long (CRT-Long; Primi, Morsanyi, Chiesi, Donati & Hamilton, 2016) and the Probabilistic Reasoning Scale (PRS; Primi, Morsanyi, Donati, Galli & Chiesi, 2017). The CRT-Long is an extended, 6-item version of the CRT (Frederick, 2005), and it measures the ability to resist intuitively compelling, but incorrect responses, and to rely on effortful processing instead.…”

“…The PRS has also been found to be related to numerical skills and cognitive abilities, and it also showed a moderate correlation with cognitive reflection (Primi et al, 2017), although no previous studies have examined the relation with deductive reasoning. Given that some items assess conditional probability reasoning, we expected a link between the PRS and conditional reasoning.…”

“…Probabilistic Reasoning Scale (PRS;Primi et al, 2017) is a 16-item multiple choice questionnaire that provides a comprehensive assessment of basic aspects of probabilistic reasoning, including basic and conditional probabilities presented in text and tables, reasoning about random sequences of events, and the ability to resist some typical fallacies and biases. Similar to the CRT-Long, the PRS consists of word problems, but the participants have to select the correct response out of 3 options, instead of generating a response.…”

Gambling-Specific Cognitions Are Not Associated With Either Abstract or Probabilistic Reasoning: A Dual Frequentist-Bayesian Analysis of Individuals With and Without Gambling Disorder

“…A second category of studies has intentionally investigated the putative associations between gambling severity (or presence of gambling disorder/problem gambling) and domain-general reasoning ( Templer et al, 1993 ; Fernández-Montalvo et al, 1999 ; Delfabbro et al, 2006 ; Lambos and Delfabbro, 2007 ; Kaare et al, 2009 ; Hodgins et al, 2012 ; Rai et al, 2014 ; Primi et al, 2017 ) in broad community or convenience samples, using regression or correlation techniques. These show that individuals with low domain-general reasoning abilities show more severe gambling problems or are in a higher risk of presenting disordered or problematic gambling, with few exceptions [( Fernández-Montalvo et al, 1999 ); in Primi et al (2017) , gambling problems’ severity was found to correlate positively with fluid intelligence, but negatively with probabilistic reasoning]. Again, however, gambling-specific cognitions were not central variables of interest.…”

An item response theory approach to constructing and evaluating brief and in‐depth financial literacy scales

“…Whereas in classical test theory measures of latent ability vary with the average difficulty of the test items, those resulting from IRT do not. IRT has been used to construct scales in several other domains such as numeracy (Weller et al ., 2013), reasoning (Primi et al ., 2017), and financial wellbeing (Consumer Finance Protection Bureau, 2015). As an in‐depth explication of IRT is beyond the scope of this report (see Embretson and Reise, 2000, for a detailed breakdown), we focus our introduction to IRT on detailing the two parameters estimated and outlining the assumptions of two‐parameter IRT models.…”