Evaluating the Consistency of Angoff-Based Cut Scores Using Subsets of Items Within a Generalizability Theory Framework

Abstract

The Angoff method requires experts to view every item on the test and make a probability judgment. This can be time consuming when there are large numbers of items on the test. In this study, a G-theory framework was used to determine if a subset of items can be used to make generalizable cut-score recommendations. Angoff ratings (i.e., probability judgments) from previously conducted standard setting studies were used first in a re-sampling study, followed by D-studies. For the re-sampling study, proportionally stratified subsets of items were extracted under various sampling and test-length conditions. The mean cut score, variance components, expected standard error (SE) around the mean cut score, and root-mean-squared deviation (RMSD) across 1,000 replications were estimated at each study condition. The SE and the RMSD decreased as the number of items increased, but this reduction tapered off after approximately 45 items. Subsequently, D-studies were performed on the same datasets. The expected SE was computed at various test lengths. Results from both studies are consistent with previous research indicating that between 40-50 items are sufficient to make generalizable cut score recommendations.

Comparison of Passing Scores Determined by The Angoff Method in Different Item Samples

“…In contrast, no significant level of difference was observed in the other 12 cases. These results comparatively support the findings of Kannan, Sgammato, Tannenbaum and Katz (2015). reported that the predicted mean Angoff MPSs did not change much for different sampling methods or different subtests.…”

“…Item reduction is recommended to be used, especially given that the standard setting processes using the Angoff and similar methods are very time consuming, very exhausting and require more cognitive effort. When the related literature is examined, it is seen that there are studies on reducing the number of items in standard setting studies using the Angoff method (Ferdous & Plake, 2005;Ferdous & Plake, 2007;Kannan, Katz, Sgammato, & Tannenbaum Katz, 2015;Plake & Impara, 2001;Smith, 2011); however, it was detected that in terms of reducing the number of items, studies which analyze the effectiveness of stratified random sampling methods (content stratified [C-SRS], difficulty stratified [D-SRS], content-difficulty stratified [CD-SRS], content-difficultydiscrimination stratified [CDD-SRS] etc.) are limited in number.…”

Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination

“…Ferdous and Plake [ 3 ] in 2005 reported that if the size of the subset was 50% or more of the entire item set, the resulting cut score was very similar to the score derived from the entire item set. Kannan et al [ 11 ] in 2015 showed through an analysis using generalizability theory that at least 40 to 50 items were required to achieve estimations with a reliability of 0.80 to 0.90.…”

Standard-setting methodology: Establishing performance standards and setting cut-scores to assist score interpretation

“…As such, in terms of best practices, establishing a test standard and cut-score involves conceptualizing it as evidence/data-based policy making that is essentially tied to test validity and establishing an evidential trail that supports that the proposed performance standards and cut-scores are not unreasonable and are reproducible and generalizable, akin to more widely accepted day-to-day scientific practices. There are very recent advances in this approach that involve adapting psychometric methods (e.g., generalizability theory) such as those described in Kannan et al (2015).…”