The Complexity of Assessing Student Work Using Comparative Judgment: The Moderating Role of Decision Accuracy

Abstract

Nowadays, comparative judgment (CJ) is used to assess competences. Judges compare two pieces of student work and judge which of both is better regarding the competence assessed. Using these pairwise comparison data, students' work is scaled according to its quality. Since student work is highly information loaded and heterogeneous of nature, this raises the question whether judges can handle this type of complex judgments? However, research into the complexity of CJ and its relation with decision accuracy is currently lacking. Therefore, this study initiates a theoretical framework on the complexity of CJ and relates it to decision accuracy. Based on this framework, two hypotheses are formulated and their plausibility is examined. The first hypothesis states that the distance between two pieces of student work on the rank-order (rank-order distance) is negatively related to experienced complexity, irrespectively of decision accuracy. In contrast, hypothesis 2 expects decision accuracy to moderate the relation between rank-order distance and experienced complexity. A negative relation is expected for accurate decisions. Meanwhile, inaccurate decisions are assumed to result in higher experienced complexity than accurate decisions, irrespective of rank-order distance. In both hypotheses, judges are expected to vary in mean experienced complexity as well as in the strength of the expected relationship between rank-order distance and experienced complexity. Using an information-theoretic approach, both hypotheses are translated into a statistical model and their relative fit is assessed. All analyses are replicated on three samples. Sample 1 and 2 comprise CJ data on the assessment of writing, while sample 3 contains pairwise comparison data on the assessment of visual arts. In all samples, results unambiguously confirm the moderating role of decision accuracy (hypothesis 2). Inaccurate decisions are experienced as more complex than accurate decisions, irrespective of rank-order distance. Meanwhile, for accurate decisions, rank-order distance is negatively related to experienced complexity. In line with expectations, differences between judges are found in mean experienced complexity and in the strength of the relationship between rankorder distance and experienced complexity. Suggestions for further theory development are formulated based on the results of this study.

Mapping Master’s Students’ Use of External Sources in Source-Based Writing in L1 and L2

“…The average ICC based on standardized scores was .78 (high reliability). We strived to reach high reliability and therefore each text in Dutch was assessed 20.1 times on average and each text in English was assessed 19.8 times (van Daal et al, 2017).…”

“…3 No students were excluded from the test based on their typing skills. 4 The pairwise comparison data were modeled according to the Bradley-Terry-Luce model, which results in a continue variable in Z-scores (Van Daal et al, 2017).…”

A meta-analysis on the reliability of comparative judgement

“…A second prominent characteristic of CJ is the number of assessors. One of the strengths of CJ is that it allows for, and even promotes, the use of multiple assessors, which increases the validity and generalizability of the assessment results (Lesterhuis et al, 2018;van Daal et al, 2017). It could also be argued, however, that engaging more assessors in the assessment leads to more differing perspectives on the particular competence under assessment, which might decrease the reliability.…”

“…A tentative hypothesis is that more representations comprise a higher range in quality, and hence a higher true variance. Due to the comparative nature of CJ, larger quality differences may facilitate the differentiation between representations (van Daal et al, 2017), leading to more consistent judgements and, hence, a higher reliability. Furthermore, more representations imply that assessors will have to make more comparisons, if the number of assessors is fixed.…”

Adaptive Pairwise Comparison for Educational Measurement

“…For each of the 2  4  10 (Algorithm  Number of Objects  Proportion of Comparisons) ¼ 80 design cells, we drew N object parameters from the standard normal distribution. We used the conventional standard normal distribution because the ASA can be applied in a wide variety of contexts, and a previous article that reported unbiased distributional properties, resulting from nonadaptive pairwise comparison, reported different standard deviations for different samples (Van Daal et al, 2017), indicating that various standard deviations may be plausible. Because the object parameter estimates have a mean of 0 as a constraint for model identification, we rescaled the object parameters to have a sample mean of 0 as well.…”