Computerized Adaptive Testing with the Zinnes and Griggs Pairwise Preference Ideal Point Model

Abstract

This article delves into a relatively unexplored area of measurement by focusing on adaptive testing with unidimensional pairwise preference items. The use of such tests is becoming more common in applied non-cognitive assessment because research suggests that this format may help to reduce certain types of rater error and response sets commonly associated with the traditional single stimulus format. Yet there have been no publications evaluating the performance of unidimensional pairwise preference adaptive or nonadaptive tests. This article therefore presents the results of a simulation study that examined scoring accuracy for three item selection algorithms (nonadaptive, adaptive-symmetric, and adaptive-asymmetric), two pool sizes (50 and 100 stimuli), two methods for pool composition (even-and over-sampling), and three test lengths (10, 20, and 40 items).

Item Selection and Exposure Control Methods for Computerized Adaptive Testing with Multidimensional Ranking Items

“…The maximum exposure rate r max can be specified based on (a) practical needs (e.g., two repetitions; Stark & Chernyshenko, ) or (b) a study showing that five repetitions of a stimulus influence participant responses ( mere exposure effect; Monahan, Murphy, & Zajonc, ). To enable an acceptably short computation time for administering an item, we sacrifice some measurement precision and restrict MFI to a lower r max (in this study, r max = .15 is likely an acceptable compromise).…”

“…In Stark and Chernyshenko's () freeze method, after a statement is presented a maximum threshold number of times, it cannot be presented again. (The multidimensional priority index, MPI, operates similarly but uses a different selection algorithm; Yao, , .)…”

Linking Methods for the Zinnes–Griggs Pairwise Preference IRT Model

“…However, psychometric advances over the last three decades have made it possible to create FC measures that provide the normative scores needed for interindividual comparisons, thus expanding potential applications (e.g., Brown & Maydeu-Olivares, 2013; Stark & Chernyshenko, 2011). Furthermore, recent studies have paid more attention to culture-specific response biases, such as extreme responding and acquiescence, which commonly distort relationships between self-report measures and outcomes of interest in cross-cultural studies (Ferrando, Anguiano-Carrasco, & Chico, 2011).…”

Adaptive Testing With Multidimensional Pairwise Preference Items

“…Yet, even with unidimensional CATs based on well-known logistic IRT models, absolute biases on the order of .3 are not uncommon with tests having 15 to 25 items, unless the pool is rich in highly discriminating items across all trait levels. Past simulation studies with unidimensional forced choice models, such as the Zinnes-Griggs unidimensional pairwise preference model, have shown absolute biases ranging from approximately .2 to .4 with adaptive tests and .3 to .6 with nonadaptive tests involving 10 to 40 pairwise preference items (e.g., Stark & Chernyshenko, 2011; Stark & Drasgow, 2002). The values are similar to what we found here with the more complex MUPP IRT model, and importantly, regression to the mean effects does not pose a problem for decision making because the correct rank order of true trait scores is generally preserved.…”