On the Statistical and Practical Limitations of Thurstonian IRT Models

Abstract

Forced-choice questionnaires have been proposed to avoid common response biases typically associated with rating scale questionnaires. To overcome ipsativity issues of trait scores obtained from classical scoring approaches of forced-choice items, advanced methods from item response theory (IRT) such as the Thurstonian IRT model have been proposed. For convenient model specification, we introduce the thurstonianIRT R package, which uses Mplus, lavaan, and Stan for model estimation. Based on practical considerations, we establish that items within one block need to be equally keyed to achieve similar social desirability, which is essential for creating forced-choice questionnaires that have the potential to resist faking intentions. According to extensive simulations, measuring up to five traits using blocks of only equally keyed items does not yield sufficiently accurate trait scores and inter-trait correlation estimates, neither for frequentist nor for Bayesian estimation methods. As a result, persons’ trait scores remain partially ipsative and, thus, do not allow for valid comparisons between persons. However, we demonstrate that trait scores based on only equally keyed blocks can be improved substantially by measuring a sizable number of traits. More specifically, in our simulations of 30 traits, scores based on only equally keyed blocks were non-ipsative and highly accurate. We conclude that in high-stakes situations where persons are motivated to give fake answers, Thurstonian IRT models should only be applied to tests measuring a sizable number of traits.

Can High-Dimensional Questionnaires Resolve the Ipsativity Issue of Forced-Choice Response Formats?

“…Additionally, the most prominent commercial FC questionnaire, the OPQ, measures -depending on the version -30 or 32 traits (Brown & Bartram, 2011;Saville et al, 1992). Sample sizes in earlier simulations were comparably high (1,000 observations in Brown & Maydeu-Olivares, 2011, and 2,000 observations in Bürkner et al, 2019). Such high sample sizes optimize the estimation of model parameters, but in practice, samples of this size are not always available.…”

“…Many for-profit test developers have adjusted their instruments accordingly and a substantial part of the literature on T-IRT has been (co)authored by for-profit test developers (e.g., Anguiano-Carrasco et al, 2015;Brown & Bartram, 2013;Brown et al, 2017;Joubert et al, 2015;Lewis, 2015;Walton et al, 2019;Watrin et al, 2019), demonstrating the practical relevance of the model. Yet a recent simulation suggests that simply changing the parameter estimation method does not lead to sufficiently reliable estimates for many FC questionnaires (Bürkner et al, 2019). When the simulated questionnaires measured five or fewer traits, the model failed to reach a satisfactory level of measurement precision in all practically relevant conditions.…”

thurstonianIRT: Thurstonian IRT Models in R

“…In addition to those general advantages, forced-choice questionnaires and corresponding IRT models specifically come with the hope of providing more valid inference in situations where participants have motivation to fake. Whether they live up to this hope remains a topic of debate (e.g., see Bürkner, Schulte, & Holling, 2019) but it is in any case necessary to provide software for fitting these statistical models both for research and practical purposes.…”

“…This probability depends on parameters related to the items under comparison as well as on parameters related to the participants' latent traits, which are assumed to be measured by the items. For more details see Brown & Maydeu-Olivares (2011) and Bürkner et al (2019). For model estimation, thurstonianIRT offers multiple backends, most notably the open source packages Stan (Carpenter et al, 2017) and lavaan (Rosseel, 2012).…”

“…For model estimation, thurstonianIRT offers multiple backends, most notably the open source packages Stan (Carpenter et al, 2017) and lavaan (Rosseel, 2012). The thurstonianIRT package was originally developed as part of a project that led to the publication of Bürkner et al (2019) but has since been developed further to fit and postprocess a more broad set of models for analysing forced-choice data. For instance, the formulation of the Thurstonian IRT model may be extended to ordinal or continuous comparative judgements, which are an active area of research facilitated by the thurstonianIRT package.…”