Comparative Judgment: An Overview

Editorial: Current perspectives on the value, teaching, learning, and assessment of design in STEM education

“…For instance, the knowledge base of designerly thinking and doing, though growing, is yet to be articulated in terms of "what" constitutes design knowledge, "how" it is constituted, "when" and "how" it is and can be acquired, and "why" it matters (Buckley et al, 2021). Furthermore, methodological frameworks for guiding, measuring, and evaluating designerly thinking, doing, and learning are in their developmental stages, indicating a need for empirical studies (Blom and Bogaers, 2020;Hartell and Buckley, 2021). The ontological perspectives of design-its nature, its purpose, and its role in learning and societal progress-are also subjects of ongoing discourse (Norström and Hallström, 2023).…”

A Review of the Valid Methodological Use of Adaptive Comparative Judgment in Technology Education Research

“…Comparative judgment (CJ), particularly adaptive comparative judgment (ACJ), is presented within the pertinent literature as auspicious in that it would appear to solve this particular disciplinary problem. The process of ACJ is described in detail by Hartell and Buckley (2021), but in brief it involves a cohort of assessors, typically referred to as "judges, " who individually make holistic pairwise comparisons on digital or digitized representations of student work which are subjected to assessment, i.e., portfolios (Kimbell, 2012;Pollitt, 2012a,b). Over a series of rounds, judges make value-laden, binary judgments on portfolios which are selected for comparison based on an adaptive sorting algorithm (Canty, 2012).…”

“…The resounding answer to these questions is "yes." ACJ has been shown to be highly reliable in each relevant study which presents reliability statistics (Kimbell, 2012;Bartholomew and Yoshikawa-Ruesch, 2018;Bartholomew and Jones, 2021) and its validity can be seen as tied to the assessors (Buckley et al, 2020a;Hartell and Buckley, 2021) with outputted misfit statistics being useful to audit or gain insight into outlying judges or portfolios (Canty, 2012). While many of the conducted studies have taken the form of mechanistic, efficacy and effectiveness studies through the use of correlational and experimental designs, the research has largely been exploratory due to the lack of a theoretical framing regarding the place of ACJ within the technology classroom.…”

Using Adaptive Comparative Judgment to Holistically Assess Creativity of Design Solutions: A Comparison of First-Year Students and Educators' Judgments

“…ACJ is a holistic assessment approach which involves intentionally and adaptively pairing two items of work which are assessed by a number of individual judges to produce a rank order of performance within a group [3], [4], [11], [12]. The intentional and adaptive pairing of items of work is driven by an algorithm [13] which pairs work based on maximizing the information gained resulting from decisions made by panellists to accelerate the achievement of a reliable rank order of performance [3], [4], [7], [11].…”