Analysing Standard Progressive Matrices (SPM-LS) with Bayesian Item Response Models

Abstract

Raven’s Standard Progressive Matrices (SPM) test and related matrix-based tests are widely applied measures of cognitive ability. Using Bayesian Item Response Theory (IRT) models, I reanalyzed data of an SPM short form proposed by Myszkowski and Storme (2018) and, at the same time, illustrate the application of these models. Results indicate that a three-parameter logistic (3PL) model is sufficient to describe participants dichotomous responses (correct vs. incorrect) while persons’ ability parameters are quite robust across IRT models of varying complexity. These conclusions are in line with the original results of Myszkowski and Storme (2018). Using Bayesian as opposed to frequentist IRT models offered advantages in the estimation of more complex (i.e., 3–4PL) IRT models and provided more sensible and robust uncertainty estimates.

A Mokken Scale Analysis of the Last Series of the Standard Progressive Matrices (SPM-LS)

“…While the SPM-LS has already been investigated using a variety of methods in this very dataset—including parametric IRT, Bayesian IRT, factor analysis, and exploratory graph analysis ( Myszkowski and Storme 2018 ; Garcia-Garzon et al 2019 ; Bürkner 2020 )—the current study proposes the first investigation of this instrument using non-parametric IRT, and more specifically Mokken Scale Analysis ( Mokken 1971 ; Mokken and Lewis 1982 ). This framework allowed to study several psychometric properties, permitting to both confirm the previous encouraging results on the SPM-LS—on dimensionality, local independence and reliability—and to investigate new properties—monotonicity and invariant item ordering.…”

“…Currently, it has only been studied in its original investigation ( Myszkowski and Storme 2018 )—which used binary and nominal IRT models—and as part of this special issue through a further investigation of its dimensionality ( Garcia-Garzon et al 2019 ). Investigations of the SPM-LS indicated that IRT models could satisfactorily fit test responses ( Bürkner 2020 ; Myszkowski and Storme 2018 ), and that the test seemed to present adequate reliability/information for abilities ranging from about 2 standard deviations below the mean—or 3 if recovering information from distractors—to 1.5 to 2 standard deviations above the mean ( Myszkowski and Storme 2018 ), in a sample of undergraduate students, suggesting that it could be more appropriate in terms of difficulty for the general population than for post-secondary students. In addition, Garcia-Garzon et al ( 2019 ) notably studied in this special issue the dimensionality of the SPM-LS using a variety of methods—Exploratory Graph Analysis (EGA), bifactor Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA).…”

“…Therefore, Mokken scaling is attractive for the reason that it focuses mainly on a test’s capacity to order persons, while allowing for more items to fulfill its requirements than Rasch models do allow. In the context of the SPM-LS, this is particularly interesting, especially as Myszkowski and Storme ( 2018 ) had to use highly parametrized models to achieve an acceptable fit, with 3- and 4-parameter models fitting much better than notably the Rasch 1-parameter model—in this special issue, Bürkner ( 2020 ) makes a similar conclusion using Bayesian IRT. Instead of increasing the number of parameters to better fit item responses—and risking overfitting and thus compromising reproducibility—Mokken scaling proposes to retain fewer (but fundamental) uses of a test: Ordering persons (for both MSA models) and items (for the Double Monotonicity model only).…”

Regularized Latent Class Analysis for Polytomous Item Responses: An Application to SPM-LS Data<strong></strong>

“…[1] composed the last twelve matrices of the Standard Progressive Matrices (SPM-LS) and argued that it could be regarded as valid indicator of general intelligence g. As part of this special issue, the SPM-LS dataset that was analyzed in [1] was reanalyzed in a series of papers applying a wide range of psychometric approaches. In particular, [2] investigated item distractor analysis with a particular focus on reliability, [3] provided additional insights due to dimensionality analysis, [4] applied the Haberman interaction model using the R package dexter, Mokken scaling was employed by [5], and, finally, [6] presented Bayesian item response modeling using the R package brms.…”

Analysis of an Intelligence Dataset

“… Storme et al ( 2019 ) later find that the reliability boosting strategy proposed in the original paper—which consisted of using nested logit models ( Suh and Bolt 2010 ) to recover information from distractor information—is useful in other contexts, by using the example on a logical reasoning test applied in a personnel selection context. Moreover, Bürkner ( 2020 ) later presents how to use his R Bayesian multilevel modeling package ( Bürkner 2017 ) in order to estimate various binary item response theory models, and compares the results with the frequentist approach used in the original paper with the item response theory package ( Chalmers 2012 ). Furthermore, Forthmann et al ( 2020 ) later proposed a new procedure that can be used to detect (or select) items that could present discriminating distractors (i.e., items for which distractor responses could be used to extract additional information).…”