A Bayesian Random Block Item Response Theory Model for Forced-Choice Formats

Abstract

Based on the framework of testlet models, the current study suggests the Bayesian random block item response theory (BRB IRT) model to fit forced-choice formats where an item block is composed of three or more items. To account for local dependence among items within a block, the BRB IRT model incorporated a random block effect into the response function and used a Markov Chain Monte Carlo procedure for simultaneous estimation of item and trait parameters. The simulation results demonstrated that the BRB IRT model performed well for the estimation of item and trait parameters and for screening those with relatively low scores on target traits. As found in the literature, the composition of item blocks was crucial for model performance; negatively keyed items were required for item blocks. The empirical application showed the performance of the BRB IRT model was equivalent to that of the Thurstonian IRT model. The potential advantage of the BRB IRT model as a base for more complex measurement models was also demonstrated by incorporating gender as a covariate into the BRB IRT model to explain response probabilities. Recommendations for the adoption of forced-choice formats were provided along with the discussion about using negatively keyed items.

manuscript

“…Since the intercept parameter of the TIRT model has a different meaning from the parameter of the 2PL-RANK model, in the next analysis, we compare only the estimation results of the discrimination  parameters, 24 trait correlations, and thetas. The correlation coefficient between the two models was 0.934 for the discrimination parameter, this value slightly higher than the results reported by Lee and Smith (2020) The scale reliabilities (Table 5) were calculated using the marginal reliability equation (Brown & Croudace 2015), and CORR and ABS were calculated for each dimension. The mean of reliabilities of 2PL-RANK is slightly higher than TIRT.…”

“…This research had some limitations. Foremost, since to verify the advantages of the stEM method in high-dimensional contexts, we only simulated 12-and 24-dimensional test scenarios, while previous simulation studies (Lee & Smith, 2020;Lee et al, 2019;Morillo et al, 2016) were limited by the time-consuming nature of traditional parameter estimation method, and most of their simulation conditions were in 10 dimensions and below. Although most of the established personality tests are high-dimensional (SHL 2018;Aon Hewitt, 2015), the performance of this model and algorithm in low-dimensional contexts needs to be further investigated if the 2PL-RANK model is to be used in lower dimensional assessment contexts(like Big Five personality scale tests).…”

“…TIRT is essentially a factor model, which can also be expressed in 2PL terms, so it also assumes a dominance process (2PL is one kind of dominance model, which assumes that the higher the level of an individual's trait, the higher the probability that he/she will answer positively to the corresponding item) as with MUPP-2PL. A similar model to the TIRT is the Bayesian random block (BRB) IRT model (Lee and Smith, 2020), which was built upon the Bayesian testlet model (Bradlow et al, 1999).…”

On the information obtainable from comparative judgments

“…Various simulation studies have demonstrated that normative scores can indeed be obtained by means of TIRT (or comparable) modeling approaches (Brown & Maydeu-Olivares, 2011;Bürkner, Schulte, & Holling, 2019;H. Lee & Smith, 2020;Schulte, Holling, & Bürkner, 2020), although satisfactory estimation accuracy cannot easily be achieved under all practically relevant conditions (see Section 1.2 for details).…”

“…On the other hand, existing research suggests that, with few exception discussed below, including unequally keyed item pairs is required to obtain normative trait scores from FC questionnaires (Brown & Maydeu-Olivares, 2011;Bürkner, Schulte, & Holling, 2019;H. Lee & Smith, 2020;Schulte, Holling, & Bürkner, 2020).…”