Oranges and Apples? Using Comparative Judgement for Reliable Briefing Paper Assessment in Simulation Games

Settembri, Pierpaolo; Gasse, Roos Van; Coertjens, Liesje; Mæyer, Sven De

doi:10.1007/978-3-319-74147-5_8

Cited by 2 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using this information, the grades for the other self-reflections are calculated using the logit scores from the rank order. 62 Another option, which has not been tested to our knowledge, would be to ask raters to indicate a provisional grade boundary (e.g., clearly pass/clearly fail/unsure) while they are assessing. This could inform grading afterwards.…”

Section: Discussionmentioning

confidence: 99%

“…59,60 To ensure criterion-referenced testing, an additional step is needed after obtaining rank order. Different options exist, such as a standard setting exercise to determine a pass/fail boundary 61 or expert grading of two performances on the scale, after which the grades for the other performances can be calculated (for example, see 62 ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improving Self-Reflection Assessment Practices: Comparative Judgment as an Alternative to Rubrics

Coertjens

Lesterhuis

Winter

et al. 2021

Teaching and Learning in Medicine

Self Cite

View full text Add to dashboard Cite

Construct. The authors aimed to investigate the utility of the comparative judgment method for assessing students' written self-reflections.Background. Medical practitioners' reflective skills are increasingly considered important and therefore included in the medical education curriculum. However, assessing students' reflective skills using rubrics does not appear to guarantee adequate inter-rater reliabilities. Recently, comparative judgment was introduced as a new method to evaluate performance assessments. This study investigates the merits and limitations of the comparative judgment method for assessing students' written self-reflections. More specifically, it examines the reliability in relation to the time spent assessing, the correlation between the scores obtained using the two methods (rubrics and comparative judgment), and, raters' perceptions of the comparative judgment method. Approach. Twenty-two self-reflections, that had previously been scored using a rubric, were assessed by a group of eight raters using comparative judgment. Two hundred comparisons were completed and a rank order was calculated.Raters' impressions were investigated using a focus group. Findings. Using comparative judgment, each self-reflection needed to be compared seven times with another self-reflection to reach a scale separation reliability of .55. The inter-rater reliability of rating (ICC, (1, k)) using rubrics was .56. The time investment required for these reliability levels in both methods was around 24 minutes. The Kendall's tau rank correlation indicated a strong correlation between the scores obtained via both methods. Raters reported that making comparisons made them evaluate the quality of self-reflections in a more nuanced way. Time investment was, however, considered 3 heavy, especially for the first comparisons. Although raters appreciated that they did not have to assign a grade to each self-reflection, the fact that the method does not automatically lead to a grade or feedback was considered a downside. Conclusions.First evidence was provided for the comparative judgment method as an alternative to using rubrics for assessing students' written self-reflections. Before comparative judgment can be implemented for summative assessment, more research is needed on the time investment required to ensure no contradictory feedback is given back to students. Moreover, as the comparative judgment method requires an additional standard setting exercise to obtain grades, more research is warranted on the merits and limitations of this method when a pass/fail approach is used.

show abstract

Section: Discussionmentioning

confidence: 99%