On the information obtainable from comparative judgments

Abstract

Personality tests employing comparative judgments have been proposed as an alternative to Likert-type rating scales. One of the main advantages of a comparative format is that it can reduce faking of responses in high-stakes situations. However, previous research has shown that it is highly difficult to obtain trait score estimates that are both faking resistant and sufficiently accurate for individual-level diagnostic decisions. With the goal of solving this problem, I study the information obtainable from comparative judgments analyzed by means of Thurstonian IRT models. First, I extend the mathematical theory of ordinal comparative judgments and corresponding models. Second, I provide optimal test designs for factor loadings of Thurstonian IRT models from both frequentist and Bayesian statistical perspectives. Third, I derive analytic upper bounds for the accuracy of trait score estimates achievable through these models. Fourth, I perform numerical experiments that complement results obtained in earlier simulation studies. The combined analytical and numerical results suggests that it is indeed possible to design personality tests using comparative judgments that can reduce faking in high-stakes situations while yielding trait scores estimates that are sufficiently accurate for individual-level diagnostic decisions. Recommendations for the practical application of comparative judgments for the measurement of personality, specifically in high-stakes situations, are given.