Standard‐Setting Methods as Measurement Processes

Nichols, Paul D.; Twing, Jon S.; Mueller, Canda D.; O’Malley, Kimberly J.

doi:10.1111/j.1745-3992.2009.00166.x

Cited by 22 publications

(9 citation statements)

References 19 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, given the large number (4900) of unique panels generated for this study and the concordance with previous results generated from real data [ 33 ], it is reasonable to believe that the findings are generalizable. Moreover, as already explained, the assumptions made in the generation of the data are grounded in educational measurement and standard settings theories and findings in practice [ 12 , 29 , 32 , 35 , 47 , 67 ]. Note that as expected from a simulation study, this study measures the quality of a model rather than analysing any observed data [ 22 ].…”

Section: Discussionmentioning

confidence: 99%

“…Standard setting is an important aspect of assessment, with the literature describing a plethora of methods. Although each has unique features, most standard setting methods use panels of expert judges to determine the cut-scores between the different performance categories [ 4 , 5 , 14 , 47 , 50 ]. Among the judge-based standard setting methods, Angoff’s method (henceforth Angoff) and variants, have been used in a range of educational settings [ 2 , 4 , 13 , 59 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Insights into the Angoff method: results from a simulation study

et al. 2016

View full text Add to dashboard Cite

BackgroundIn standard setting techniques involving panels of judges, the attributes of judges may affect the cut-scores. This simulation study modelled the effect of the number of judges and test items, as well as the impact of judges’ attributes such as accuracy, stringency and influence on others on the precision of the cut-scores.MethodsForty nine combinations of Angoff panels (N = 5, 10, 15, 20, 30, 50, and 80) and test items (n = 5, 10, 15, 20, 30, 50, and 80) were simulated. Each combination was simulated 100 times (in total 4,900 simulations). The simulation was of judges attributes: stringency, accuracy and leadership. Impact of judges attributes, number of judges, number of test items and Angoff’s second (compared to the first) round on the precision of a panel’s cut-score was measured by the deviation of the panel’s cut-score from the cut-score’s true value.ResultsFindings from 4900 simulated panels supported Angoff being both reliable and valid. Unless the number of test items is small, panels of around 15 judges with mixed levels of expertise provide the most precise estimates. Furthermore, if test data were not presented, a second round of decision-making, as used in the modified Angoff, adds little to precision. A panel which has only experts or only non-experts yields a cut-score which is less precise than a cut-score yielded by a mixed-expertise panel, suggesting that optimal composition of an Angoff panel should include a range of judges with diverse expertise and stringency.ConclusionsSimulations aim to improve our understanding of the models assessed but they do not describe natural phenomena as they do not use observed data. While the simulations undertaken in this study help clarify how to set cut-scores defensibly, it is essential to confirm these theories in practice.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Insights into the Angoff method: results from a simulation study

et al. 2016

View full text Add to dashboard Cite

show abstract

“…The candidate's true level of ability will be above or below the categorisation threshold, it is just the decision makers who are uncertain. The individual candidate score and categorisation threshold score are estimates, as there is measurement error in both (Nichols et al 2010). So the difficulties in categorisation reflect the error in the difference between these two estimates.…”

Section: Dear Sirmentioning

confidence: 99%

Letters to the Editor

2010

Medical Teacher

View full text Add to dashboard Cite

“…If such data (the examinees’ ‘true’ ability) were available, there would be no need to set standards and cut-scores. Consequently, the quality of standard setting methods is commonly measured by the level of subjective agreement among judges, the reliability of the results, or the error of measurement of the yielded cut-scores [ 22 – 29 ].…”

Section: Introductionmentioning

confidence: 99%

Cut-scores revisited: feasibility of a new method for group standard setting

et al. 2018

View full text Add to dashboard Cite

BackgroundStandard setting is one of the most contentious topics in educational measurement. Commonly-used methods all have well reported limitations. To date, there is not conclusive evidence suggesting which standard setting method yields the highest validity.MethodsThe method described and piloted in this study asked expert judges to estimate the scores on a real MCQ examination that they consider indicated a clear pass, clear fail, and pass mark for the examination as a whole. The mean and SD of the judges responses to these estimates, Z scores and confidence intervals were used to derive the cut-score and the confidence in it.ResultsIn this example the new method’s cut-score was higher than the judges’ estimate. The method also yielded estimates of statistical error which determine the range of the acceptable cut-score and the estimated level of confidence one may have in the accuracy of that cut-score.ConclusionsThis new standard-setting method offers some advances, and possibly advantages, in that the decisions being asked of judges are based on firmer constructs, and it takes into account variation among judges.

show abstract

Standard‐Setting Methods as Measurement Processes

Cited by 22 publications

References 19 publications

Insights into the Angoff method: results from a simulation study

Insights into the Angoff method: results from a simulation study

Letters to the Editor

Cut-scores revisited: feasibility of a new method for group standard setting

Contact Info

Product

Resources

About