The method of Adaptive Comparative Judgement

Pollitt, Alastair

doi:10.1080/0969594x.2012.665354

Cited by 140 publications

(183 citation statements)

References 12 publications

Supporting

Mentioning

162

Contrasting

Order By: Relevance

“…(Pollitt, 2004, p. 6) Following the theoretical development of the ACJ process, a grading engine was commercialized by TAG Assessment under the name CompareAssess. Using a complex algorithm, which has been validated repeatedly and used on thousands of student artifacts (Pollitt, 2004(Pollitt, , 2012, CompareAssess combines rankings from a panel of judges to assign a final rank order to each artifact. In the CompareAssess engine, each artifact is compared with other artifacts by randomly assigned graders until a specified reliability requirement has been met.…”

Section: Adaptive Comparative Judgmentmentioning

confidence: 99%

Relationships Between Access to Mobile Devices, Student Self-Directed Learning, and Achievement

Bartholomew¹,

Reeve²,

Veon³

et al. 2017

Journal of Technology Education

View full text Add to dashboard Cite

Today's students are growing up in a world of constant connectivity, instant information, and ever-changing technological advancements. The increasingly ubiquitous nature of mobile devices among K-12 students has led many to argue for and against the inclusion of these devices in K-12 classrooms. Arguments in favor cite instant access to information and collaboration with others as positive affordances that enable student self-directed learning.In this study, 706 middle school students from 18 technology and engineering education classes worked in groups of 2-3 to complete an openended engineering design challenge. Students completed design portfolios and constructed prototypes in response to the design challenge. Classes were divided with some allowing access to mobile devices during the study and others not allowing access. Additionally, randomly assigned classes completed the design portfolio electronically, and others completed the portfolio on paper. Final student portfolios and products were assessed and assigned a rank order using a method of assessment called adaptive comparative judgment. Thirty student interviews were conducted as well as 6 teacher interviews. Statistical analyses between student access, portfolio type, student self-directed learning, and student achievement were conducted. Findings showed that student self-directed learning was independent of mobile device access during the study. Mobile device access was significantly correlated with higher student scores on the design portfolio, but mobile device access was independent of student scores on design products.

show abstract

Section: Adaptive Comparative Judgmentmentioning

confidence: 99%

Relationships Between Access to Mobile Devices, Student Self-Directed Learning, and Achievement

Bartholomew¹,

Reeve²,

Veon³

et al. 2017

Journal of Technology Education

View full text Add to dashboard Cite

show abstract

“…The comparative pairs approach to marking requires assessors to select a 'winner' between a pair of performances, and repeat this process for many pairs, with the results analysed using a Rasch model for dichotomous data (Pollitt, 2012). Whereas Pollitt (2004) describes the comparative pairs method as "intrinsically more valid", he believes that without Information and Communications Technology (ICT) support it has not been feasible to apply due to time and cost constraints.…”

Section: Methods Of Markingmentioning

confidence: 99%

“…For the third phase, the rubrics were sent to the Willock Information Systems company and made available through their online assessment module for teachers and assessors for analytical marking. For comparative pairs marking the online marking tool called the Adaptive Comparative Judgement System (ACJS) (Pollitt, 2012), developed by TAG Learning for the e-scape research using MAPS, was used. Student responses (typed and oral recording) in digital form were downloaded from the Willock Information Systems website and uploaded to MAPS.…”

Section: Marking Criteria Tools and Assessorsmentioning

confidence: 99%

Computer-based oral exams in Italian language studies

Newhouse

Cooper

2013

ReCALL

View full text Add to dashboard Cite

In this paper we report on one component of a three-year study into the use of digital technologies for summative performance assessment in senior secondary courses in Western Australia. One of the courses was Italian Studies, which had an oral communication outcome externally assessed with an oral performance for which students travelled to a central location and undertook an interview with two assessors. Apart from the logistical difficulties for both students and the organising body, this method didn't leave an enduring record of the process, and raised questions about the reliability of the assessment. Over the three years of this study, we tried several approaches to using digital technology to assess oral performance, including a portfolio of sub-tasks leading up to a video-recorded oral presentation, a computer-based exam, a video recorded interview, and an online exam that included oral audio-recordings. For each of the years online marking tools supported two methods of drawing inferences about student performance from the representations: the more traditional analytical method and the comparative pairs method. Rasch analysis of the results of the two methods showed that both were at an acceptable level of reliability. Overall students and teachers reported that they liked using audiovisual recordings and online performance tasks for revision but not for summative assessment. The study also demonstrated that the scores from externally marked computer-based oral tasks carried out in class time correlated highly with the scores from traditional face-to-face recorded interviews. Therefore, online assessment of oral performance appears to be an equally effective way to facilitate assessment when compared with traditional methods and offers other affordances, such as convenience and access from a variety of locations, as well as providing an enduring record of student output.

show abstract

“…Pollitt (2012) describes in detail how the ACJS combines all the main processes involved in scoring using the comparative pairs method including generating the pairs of portfolios for each assessor to judge, provided a facility for recording that judgement and maintaining assessor notes, and calculating scores and associated reliability coefficients using Rasch modelling. This meant that assessors only needed to judge pairs until an acceptable level of reliability was attained.…”

Section: Analytical and Comparative Pairs Scoringmentioning

confidence: 99%

“…This has been impractical on a large scale until the recent development of online systems to facilitate the processes, such as the system used by the e-Scape project (Kimbell, Wheeler, Miller, & Pollitt, 2007). The method relies on the use of Rasch statistical modeling to generate scores on an interval scale, and has delivered highly reliable sets of scores (Pollitt, 2012), including in our own research (Newhouse,2010).…”

Section: Introductionmentioning

confidence: 99%

Digitizing practical production work for high-stakes assessments / La numérisation de travaux pratiques de production pour les évaluations à enjeux élevés / La numérisation de travaux pratiques de production pour les évaluations à enjeux élevés

Newhouse

Tarricone

2014

CJLT / RCAT

View full text Add to dashboard Cite

High-stakes external assessment for practical courses is fraught with problems impacting on the manageability, validity and reliability of scoring. Alternative approaches to assessment using digital technologies have the potential to address these problems. This paper describes a study that investigated the use of these technologies to create and submit digital representations of practical production work and forms of creative expression for summative high-stakes assessment. The study set out to determine the feasibility of students creating and submitting these digital representations for assessment and to identify which of analytical or comparative pairs scoring generated the more reliable scores. This paper proposes that scoring digital representations of creative practical work submitted by students is a viable alternative to traditional approaches to assessment.Key words: Digital portfolio, practical assessment, comparative pairs scoring, analytical scoring Résumé L'évaluation externe à enjeux élevés dans les cours pratiques se heurte à des problèmes qui se répercutent sur la gestion, la validité et la fiabilité de la notation. Des approches différentes de l'évaluation utilisant des technologies numériques ont le potentiel de remédier à ces problèmes. Cet article décrit une étude consacrée à l'utilisation de ces technologies pour créer et soumettre des représentations numériques de travaux pratiques de production et de création pour une évaluation sommative à enjeux élevés. L'étude visait à déterminer si la création de ces représentations numériques par les étudiants et leur soumission pour évaluation étaient réalisables. Elle visait aussi à identifier quel système de notation de groupe, analytique ou comparatif, générait les scores les plus fiables. Cet article soutient que noter les représentations CJLT/RCAT Vol. 40 (2) Digitizing practical production work for high--stakes assessments 2 numériques de travaux pratiques soumis par les étudiants offre un choix viable aux approches traditionnelles d'évaluation.

show abstract

The method of Adaptive Comparative Judgement

Cited by 140 publications

References 12 publications

Relationships Between Access to Mobile Devices, Student Self-Directed Learning, and Achievement

Relationships Between Access to Mobile Devices, Student Self-Directed Learning, and Achievement

Computer-based oral exams in Italian language studies

Digitizing practical production work for high-stakes assessments / La numérisation de travaux pratiques de production pour les évaluations à enjeux élevés / La numérisation de travaux pratiques de production pour les évaluations à enjeux élevés

Contact Info

Product

Resources

About