GGUM-RANK Statement and Person Parameter Estimation With Multidimensional Forced Choice Triplets

Abstract

Historically, multidimensional forced choice (MFC) measures have been criticized because conventional scoring methods can lead to ipsativity problems that render scores unsuitable for interindividual comparisons. However, with the recent advent of item response theory (IRT) scoring methods that yield normative information, MFC measures are surging in popularity and becoming important components in high-stake evaluation settings. This article aims to add to burgeoning methodological advances in MFC measurement by focusing on statement and person parameter recovery for the GGUM-RANK (generalized graded unfolding-RANK) IRT model. Markov chain Monte Carlo (MCMC) algorithm was developed for estimating GGUM-RANK statement and person parameters directly from MFC rank responses. In simulation studies, it was examined that how the psychometric properties of statements composing MFC items, test length, and sample size influenced statement and person parameter estimation; and it was explored for the benefits of measurement using MFC triplets relative to pairs. To demonstrate this methodology, an empirical validity study was then conducted using an MFC triplet personality measure. The results and implications of these studies for future research and practice are discussed.

Item Selection and Exposure Control Methods for Computerized Adaptive Testing with Multidimensional Ranking Items

“…Wang & Chen, ). Specifically, a ranking item in this study is composed of a group of statements, and each statement loads on only one dimension (e.g., a 10‐dimensional test uses triplet ranking items; each item consists of three statements involving three of 10 dimensions; Lee, Joo, Stark, & Chernyshenko, ). Unlike cognitive tests, noncognitive tests often have many dimensions and items (e.g., SNAP‐2 has 15 dimensions and 390 items; Clark, Simms, Wu, & Casillas, ), requiring long wait times before an item is selected.…”

Though Forced, Still Valid: Psychometric Equivalence of Forced-Choice and Single-Statement Measures

“…To our knowledge, however, only a few studies have done this. Among those few studies, most have examined the equivalence of dominance-model-based SS and FC formats and found generally supportive evidence (Brown & Maydeu-Olivares, 2011Guenole, Brown, & Cooper, 2016;Lee, Lee, &Stark, 2018). However, as mentioned above, evidence has been accumulating that shows ideal point models more accurately capture the response processes underlying various psychological measures.…”

Item Desirability Matching in Forced-choice Test Construction

“…Our focus on pairwise similarity was thus natural to provide a direct comparison with the mean difference index. Recently, a format consisting of three items (i.e., triads or triplets) per forced-choice block has been gaining in popularity (e.g., Guenole et al, 2018;Lee et al, 2019;;Murano et al, 2020;Ng et al, 2020;Walton et al, 2020;Watrin et al, 2019;, because it seems to provide an optimal balance between the information gained and the cognitive burden placed on test takers (Brown & Maydeu-Olivares, 2013). That being said, future research should also consider evaluating indices of similarity involving more than two items.…”

“…Based on the discussion so far, it is clear that when constructing a forced-choice measure resistant to SDR, it is essential to identify item-combinations for which differences in desirability evaluations between items are sufficiently similar across test takers. Typically, item desirability matching relies on empirically obtained item desirability ratings (e.g., Chernyshenko et al, 2009;Christiansen et al, 2005;Converse et al, 2010;Drasgow et al, 2012;Guenole et al, 2018;Lee et al, 2019;Naemi et al, 2014;Usami et al, 2016;Vasilopoulos et al, 2006;Watrin et al, 2019). In one common approach, a "desirability sample" is asked to explicitly rate desirability of each item under consideration (e.g., Chernyshenko et al, 2009;Christiansen et al, 2005;Usami et al, 2016).…”