Adaptive Pairwise Comparison for Educational Measurement

Crompvoets, Elise Anne Victoire; Béguin, Anton; Sijtsma, Klaas

doi:10.3102/1076998619890589

Cited by 16 publications

(45 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous studies provide varying recommendations for the number of comparisons per object required for reliable measurement. The recommendations range from = 9 (Pollitt, 2012), to = 12 (Verhavert et al, 2019, to = 20 based on average reliability and = 30 based on the lower bound of a 68% confidence interval for reliability equal to .80 (Crompvoets et al, 2020 Fourth, we varied the number of raters, using two, three, and five raters. We assumed that all raters performed an equal number of comparisons.…”

Section: Methodsmentioning

confidence: 99%

“…Fifth, we varied rater agreement as either perfectly model compliant, where the comparisons of all raters are based on the preference probabilities in the generating model, or imperfectly model compliant, where one rater does not comply perfectly with the generating model. Although perfect model compliance for all raters is unlikely, we included this condition because it facilitates comparison with previous research (Crompvoets et al, 2020). We simulated less than perfect model compliance by adding a noise component to the true values of the latent variable for one of the raters:…”

Section: Methodsmentioning

confidence: 99%

“…This procedure, starting with drawing 30 object parameters in all conditions, was repeated 400 times. The number of repetitions was based on a previous simulation study (Crompvoets et al, 2020). For both data simulation and data analysis, Thurstone's probit model was used.…”

Section: Methodsmentioning

confidence: 99%

“…We used the uncertainty of the parameters and the reliability of the scale as evaluation criteria. These are the same evaluation criteria as Crompvoets et al (2020).…”

Section: Methodsmentioning

confidence: 99%

“…However, the initial adaptive algorithms did not sufficiently take the uncertainty of the object parameters into account, resulting in biased estimates (Bramley, 2015;Bramley & Vitello, 2018). Crompvoets, Béguin, & Sijtsma (2020) proposed an adaptive selection algorithm that reduced standard errors of parameter estimates, but did not improve reliability or rank order accuracy compared with a commonly-used semi-random algorithm. In this study, we build on the research of Crompvoets et al (2020), proposing a new Bayesian selection algorithm (BSA) to obtain both smaller standard errors of parameter estimates and higher reliability compared with a commonly-used semi-random algorithm and testing it in a simulation study that builds on their simulation design and evaluation criteria.…”

Section: Pairwise Comparison Using a Bayesian Selection Algorithm: Efficient Holistic Measurementmentioning

confidence: 99%

See 4 more Smart Citations

Pairwise comparison using a Bayesian selection algorithm: Efficient holistic measurement

Eav¹,

Aa²,

Sijtsma³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The method of pairwise comparison has been used in a wide range of contexts. In educational measurement, many pairwise comparisons are required for reliable measurement when the comparisons to be performed are selected using the commonly-used semi-random selection algorithm (SSA). We proposed a Bayesian selection algorithm (BSA) to obtain smaller standard errors of parameter estimates and higher reliability compared with the SSA, and we evaluated the performance of these algorithms in a simulation study. We conclude that 1) the BSA should be preferred to the SSA, 2) the number of comparisons required for reliable measurement depends on the object variance, and 3) the Scale Separation Reliability (SSR) may systematically overestimate reliability even when the SSA is used.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

“…We used the uncertainty of the parameters and the reliability of the scale as evaluation criteria. These are the same evaluation criteria as Crompvoets et al (2020).…”

Section: Methodsmentioning

confidence: 99%

Section: Pairwise Comparison Using a Bayesian Selection Algorithm: Efficient Holistic Measurementmentioning

confidence: 99%

See 3 more Smart Citations

Pairwise comparison using a Bayesian selection algorithm: Efficient holistic measurement

Eav¹,

Aa²,

Sijtsma³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

CEFR Labelling and Assessment Services

Breuker¹

2022

European Language Grid

View full text Add to dashboard Cite

Our pilot project aims to develop a set of text collections and annotation tools to facilitate the creation of datasets (corpora) for the development of AI classification models. These classification models can automatically assess a text’s reading difficulty on the levels described by the Common European Framework of Reference (CEFR). The ability to accurately and consistently assess the readability level of texts is crucial to authors and (language) teachers. It allows them to more easily create and discover content that meets the needs of students with different backgrounds and skill levels. Also, in the public sector using plain language in written communication is becoming increasingly important to ensure citizens can easily access and comprehend government information. EDIA already provides automated readability assessment services (available as APIs and an online authoring tool) for the CEFR in English. Support for Dutch, German and Spanish are added as part of this project. Using the infrastructure developed in this project the effort for creating high quality datasets for additional languages is lowered significantly. The tools and datasets are deployed through the European Language Grid. The project is scheduled to be completed in the second quarter of 2022.

show abstract

Examining the reliability of Adaptive Comparative Judgement (ACJ) as an assessment tool in educational settings

Kimbell

2021

Int J Technol Des Educ

View full text Add to dashboard Cite

Conventional approaches to assessment involve teachers and examiners judging the quality of learners work by reference to lists of criteria or other ‘outcome’ statements. This paper explores a quite different method of assessment using ‘Adaptive Comparative Judgement’ (ACJ) that was developed within a research project at Goldsmiths University of London between 2004 and 2010. The method was developed into a tool that enabled judges to distinguish better/worse performances not by allocating numbers through mark schemes, but rather by direct, holistic, judgement. The tool was successfully deployed through a series of national and international research and development exercises. But game-changing innovations are never flaw-less first time out (Golley, Jet: Frank Whittle and the Invention of the Jet Engine, Datum Publishing, Liphook Hampshire, 2009; Dyson, Against the odds: an autobiography, Texere Publishing, Knutsford Cheshire, 2001) and a series of careful investigations resulted in a problem being identified within the workings of ACJ (Bramley, Investigating the reliability of Adaptive Comparative Judgment, Cambridge Assessment Research Report, UK, Cambridge, 2015). The issue was with the ‘adaptive’ component of the algorithm that, under certain conditions, appeared to exaggerate the reliability statistic. The problem was ‘worked’ by the software company running ACJ and a solution found. This paper reports the whole sequence of events—from the original innovation, through deployment, the emergent problem, and the resulting solution that was published at an international conference (Rangel Smith and Lynch in: PATT36 International Conference. Research & Practice in Technology Education: Perspectives on Human Capacity and Development, 2018) and subsequently deployed within a modified ACJ algorithm.

show abstract

Adaptive Pairwise Comparison for Educational Measurement

Cited by 16 publications

References 22 publications

Pairwise comparison using a Bayesian selection algorithm: Efficient holistic measurement

Pairwise comparison using a Bayesian selection algorithm: Efficient holistic measurement

CEFR Labelling and Assessment Services

Examining the reliability of Adaptive Comparative Judgement (ACJ) as an assessment tool in educational settings

Contact Info

Product

Resources

About