Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study

Abstract

Purpose: The percent Angoff (PA) method has been recommended as a reliable method to set the cut score instead of a fixed cut point of 60% in the Korean Medical Licensing Examination (KMLE). The yes/no Angoff (YNA) method, which is easy for panelists to judge, can be considered as an alternative because the KMLE has many items to evaluate. This study aimed to compare the cut score and the reliability depending on whether the PA or the YNA standard-setting method was used in the KMLE.Methods: The materials were the open-access PA data of the KMLE. The PA data were converted to YNA data in 5 categories, in which the probabilities for a “yes” decision by panelists were 50%, 60%, 70%, 80%, and 90%. SPSS for descriptive analysis and G-string for generalizability theory were used to present the results.Results: The PA method and the YNA method counting 60% as “yes,” estimated similar cut scores. Those cut scores were deemed acceptable based on the results of the Hofstee method. The highest reliability coefficients estimated by the generalizability test were from the PA method and the YNA method, with probabilities of 70%, 80%, 60%, and 50% for deciding “yes,” in descending order. The panelist’s specialty was the main cause of the error variance. The error size was similar regardless of the standard-setting method.Conclusion: The above results showed that the PA method was more reliable than the YNA method in estimating the cut score of the KMLE. However, the YNA method with a 60% probability for deciding “yes” also can be used as a substitute for the PA method in estimating the cut score of the KMLE.

Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study

“…Specifically, among the variations of the Angoff method, we adopted the yes/no Angoff method that requires judges to make dichotomous categorical decisions. The yes/no Angoff method has an advantage because it places less cognitive burden than estimating probabilities [ 7 ].…”

“…Recently, in medical education, a simulation study examined the possibility of using the yes/no Angoff instead of the percent Angoff method for KMLE. Although the authors concluded the results of the yes/no Angoff are less reliable, the study was limited by its use of hypothetical yes/no Angoff data converted from percent Angoff data [ 7 ].…”

Issues in the 3rd year of the COVID-19 pandemic, including computer-based testing, study design, ChatGPT, journal metrics, and appreciation to reviewers

“…Kim et al [ 2 ] suggested that acceptable cut scores for the KMLE could be derived and proposed independently using the yes/no Angoff and Hostee methods. Park [ 3 ] recommended using the yes/no Angoff method with a 60% probability for deciding “yes” as a substitute for the percent Angoff method in estimating the cutoff score of the KMLE. According to this theoretical background, it is time to consider standard setting for health professions licensing examinations instead of using a 60% correct answer rate as the threshold for passing.…”