The validity and value of peer assessment using adaptive comparative judgement in design driven practical education

Abstract

This paper presents the response of the technology teacher education programmes at the University of Limerick to the assessment challenge created by the shift in philosophy of the Irish national curriculum from a craft-based focus to design-driven education. This study observes two first year modules of the undergraduate programmes that focused on the development of subject knowledge and practical craft skills. Broadening the educational experience and perspective of students to include design based aptitudes demanded a clear aligning of educational approaches with learning outcomes. As design is a complex iterative learning process it requires a dynamic assessment tool to facilitate and capture the process. Considering the critical role of assessment in the learning process, the study explored the relevance of individual student-defined assessment criteria and the validity of holistic professional judgement in assessing capability within a design activity. The kernel of the paper centres on the capacity of assessment criteria to change in response to how students align their work with evidence of capability. The approach also supported peer assessment, where student-generated performance ranks provided an insight into not only how effectively they evidenced capability but also to what extent their peers valued it. The study investigated the performance of 137 undergraduate teachers during an activity focusing on the development of design, processing and craft skills. The study validates the use of adaptive comparative judgement as a model of assessment by identifying a moderate to strong relationship with performance scores obtained by two different methods of assessment. The findings also present evidence of capability beyond the traditional measures. Level of engagement, diversity, and problem solving were also identified as significant results of the approach taken. The strength of this paper centres on the capacity of student-defined criterion assessment to evidence learning, and concludes by presenting a valid and reliable holistic assessment supported by comparative judgements.

Peer assessment using comparative and absolute judgement

“…The successful replication of Jones and Alcock's main findings for secondary school students would provide support as to the generality of the approach. Given the successful use of comparative judgement in a variety of educational contexts (Bramley, 2007;Heldsinger & Humphry, 2013;Kimbell, 2012;Seery, Canty & Phelan, 2012) we predicted that Jones and Alcock's results would be replicated.…”

“…However, one possible limitation to this approach is that comparative judgement does not necessarily provide descriptive feedback to students. Comparative judgement studies have provided peer assessors with the opportunity to provide a text-based comment when making each judgement (Seery et al, 2012). However, in our experience that slows the judgement process and produces comments that lack insight.…”

Peer assessment without assessment criteria

“…An early application of CJ to educational assessment was in standards monitoring across equivalent forms of examination papers (Bramley, Bell and Pollitt 1998). More recent studies have tested CJ in a variety of assessment contexts (Kimbell 2012), including school mathematics (Jones, Swan and Pollitt submitted) and peer assessment (Seery, Canty and Phelan 2012).…”

“…First, expert and peer assessments are often conducted using different methods or different sets of criteria (e.g. Chang et al 2011;Seery, Canty and Phelan 2012). In our study, this was not problematic because explicit judging criteria were not used; validity is simply operationalised as the extent to which students understand the global construct "conceptual understanding" in the same way as experts.…”

The Complexity of Assessing Student Work Using Comparative Judgment: The Moderating Role of Decision Accuracy

“…inTrODUcTiOn Recently, comparative judgment (CJ) was introduced into the field of educational measurement to assess competences, such as mathematical understanding (Jones et al, 2015), geography (Whitehouse and Pollitt, 2012), design and technology (Seery et al, 2012), and writing (Pollitt, 2012a;van Daal et al, 2016). Judges are presented with two pieces of student work (e.g., essays or portfolio's) and are asked to judge which piece shows more of the competence assessed.…”