Pairwise comparison using a Bayesian selection algorithm: Efficient holistic measurement

Abstract

The method of pairwise comparison has been used in a wide range of contexts. In educational measurement, many pairwise comparisons are required for reliable measurement when the comparisons to be performed are selected using the commonly-used semi-random selection algorithm (SSA). We proposed a Bayesian selection algorithm (BSA) to obtain smaller standard errors of parameter estimates and higher reliability compared with the SSA, and we evaluated the performance of these algorithms in a simulation study. We conclude that 1) the BSA should be preferred to the SSA, 2) the number of comparisons required for reliable measurement depends on the object variance, and 3) the Scale Separation Reliability (SSR) may systematically overestimate reliability even when the SSA is used.

On the bias and stability of the results of comparative judgment

“…In addition to the claim of higher validity of comparative judgment, Pollitt (2012) claimed that comparative judgment also results in higher reliability compared to using rubrics or criteria lists. However, later research has shown that this claim is likely to be too optimistic (e.g., Bramley, 2015;Bramley & Vitello, 2018;Crompvoets, Béguin, & Sijtsma, 2020;2021).…”

“…To explain why Pollitt's (2012) claim is too optimistic, we first define two types of reliability in the context of comparative judgment: the benchmark reliability (Crompvoets et al, 2020(Crompvoets et al, , 2021 and the Scale Separation Reliability (SSR; e.g., Bramley, 2015;Crompvoets et al, 2020). Both forms of reliability are based on parameters of the Bradley-Terry-Luce (BTL; Bradley & Terry, 1952;Luce, 1959) model.…”

“…To the best of our knowledge, no one has thoroughly investigated and reported the positive bias of the SSR. Previous research that reported the bias of the SSR has stopped at the conclusion that the SSR was biased (Bramley, 2015;Bramley & Vitello, 2018) or has only led to speculations about the meaning of the bias due to either adaptive pair selection (Crompvoets et al, 2020), different rater probabilities, or small true variances (Crompvoets et al, 2021).…”

On the Bias and Stability of the Results of Comparative Judgment

“…In addition to the claim of higher validity of comparative judgment, Pollitt (2012) claimed that comparative judgment also results in higher reliability compared to using rubrics or criteria lists. However, later research has shown that this claim is likely to be too optimistic for the reported numbers of comparisons per item (e.g., Bramley, 2015;Bramley and Vitello, 2018;Crompvoets et al, 2020;Crompvoets et al, 2021), and that the extent to which high reliability that can be obtained using comparative judgment is limited (Verhavert et al, 2019).…”

“…To explain why Pollitt's (2012) claim is too optimistic, we first define two types of reliability in the context of comparative judgment: the benchmark reliability (Crompvoets et al, 2020(Crompvoets et al, , 2021 and the Scale Separation Reliability (SSR; e.g., Bramley, 2015;Crompvoets et al, 2020). Both forms of reliability are based on parameters of the Bradley-Terry-Luce (BTL; Bradley and Terry, 1952;Luce, 1959) model.…”