On the Information Obtainable from Comparative Judgments

Abstract

Personality tests employing comparative judgments have been proposed as an alternative to Likert-type rating scales. One of the main advantages of a comparative format is that it can reduce faking of responses in high-stakes situations. However, previous research has shown that it is highly difficult to obtain trait score estimates that are both faking resistant and sufficiently accurate for individual-level diagnostic decisions. With the goal of contributing to a solution, I study the information obtainable from comparative judgments analyzed by means of Thurstonian IRT models. First, I extend the mathematical theory of ordinal comparative judgments and corresponding models. Second, I provide optimal test designs for Thurstonian IRT models that maximize the accuracy of people’s trait score estimates from both frequentist and Bayesian statistical perspectives. Third, I derive analytic upper bounds for the accuracy of these trait estimates achievable through ordinal Thurstonian IRT models. Fourth, I perform numerical experiments that complement results obtained in earlier simulation studies. The combined analytical and numerical results suggest that it is indeed possible to design personality tests using comparative judgments that yield trait scores estimates sufficiently accurate for individual-level diagnostic decisions, while reducing faking in high-stakes situations. Recommendations for the practical application of comparative judgments for the measurement of personality, specifically in high-stakes situations, are given.

How do applicants fake? A response process model of faking on multidimensional forced‐choice personality assessments

“…Future research should aim at creating full control over item desirability. (Bürkner, 2022). In this case, we propose to write items that do not immediately appear undesirable although they measure the "negative" end of a trait (e.g., I speak up when people are wrong is negatively keyed to Agreeableness but is not generally an undesirable behavior).…”

Measuring Personality When Stakes Are High: Are Graded Paired Comparisons a More Reliable Alternative to Traditional Forced-Choice Methods?

“…Studies comparing graded response designs and binary FC scales suggest that the former can achieve higher reliabilities than the latter (Brown & Maydeu-Olivares, 2018;Bürkner, 2022). Specifically, Bürkner (2022) mathematically derives upper bounds regarding which reliabilities can be achieved with GPCs and binary FC scales, respectively.…”

“…Consequently, they can only be validly interpreted within one person (e.g., Person A is more extraverted than conscientious). However, inter-individual comparisons (e.g., Person A is more extraverted than Person B) must not be made because they require normative (i.e., absolute) trait scores, while ipsative scores provide only relative information on trait scores (Brown & Maydeu-Olivares, 2011, 2013, 2018Bürkner, 2022;Clemans, 1966;Frick et al, 2021).…”

“…with β pt as the trait score of person p on trait t, λ i as the factor loading of item i, and ε i as the corresponding item intercept. As is common practice when fitting TIRT models (Brown & Maydeu-Olivares, 2011;Bürkner et al, 2019;Bürkner, 2022), we assume that each item i loads on one latent trait t only. The difference in the utilities u pi of two paired items i 1 and i 2 determines the latent response y pn of person p on item pair n. The latent response can be modeled as follows:…”

“…with i 1 [n] and i 2 [n] being the first and second item in the n th paired comparison loading on trait t 1 [n] and t 2 [n], respectively. In accordance with Bürkner (2022), the following model-specifications are assumed: The intercepts ε i are distributed N (0, ψ i ) with the corresponding variance ψ 2 i being the uniqueness of the i th item. Furthermore, we set ψ 2 i = 1 − λ 2 i to yield standardized factor loadings.…”