Comparison of results between modified-Angoff and bookmark methods for estimating cut score of the Korean medical licensing examination

Abstract

PurposeThe purpose of this study was to apply alternative standard setting methods for the Korean Medical Licensing Examination (KMLE), a criterion-referenced written examination, and to compare them to the conventional cut score used on the KMLE.MethodsThe process and results of criterion-referenced standard settings (i.e., the modified-Angoff and bookmark methods) were evaluated. The ratio of passing and failing examinees determined using these alternative standard setting methods was compared to the results of the conventional criteria. Additionally, the external, internal and procedural evaluation of these methods were reviewed.ResultsThe modified-Angoff method yielded the highest cut score, followed sequentially by the conventional method and the bookmark method. The classification agreement between the modified-Angoff and bookmark methods was 0.720 measured by Cohen’s κ coefficient. The intra-panelist classification consistency of modified-Angoff method was higher than bookmark method. However, the inter-panelist classification consistency was vice versa. The standard setting panelists’ survey results showed that the procedures of both methods were satisfactory, but panelists had more confidence in the results of the modified-Angoff method.ConclusionThe modified-Angoff method showed results that were more similar to those of the conventional method. Both new methods showed very high concordance with the conventional method, as well as with each other. The modified-Angoff method was considered feasible for adoption on the KMLE. The standard setting panelists responded positively to the modified-Angoff method in terms of its practical applicability, despite certain advantages of the bookmark method.

Using the Angoff method to set a standard on mock exams for the Korean Nursing Licensing Examination

“…The panel members who participated in the national exam for nurses as item developers had a very positive response to the application of the expert rating method (i.e., the Angoff method), which was similar to the results of previous studies of medical doctors, medical recorders, and radiologists [13,14].…”

“…The average score for whether the definition of the minimum competency was clear was 4.69 points, with response frequencies of 'agree' and 'strongly agree' of 31.25% Mean transformed into a score out of 100. and 68.75%, respectively. In a previous study of a similar process for medical doctors, the panel responded that they had difficulties defining the minimum competency, and that the definition of the minimum competency did not help greatly in the evaluation [13]. However, in this study, a consensus on the minimum competency was established smoothly.…”

“…In previous studies of national tests of medical doctors, medical recorders, and radiologists, the cut scores derived by applying the Angoff method were all higher than the existing reference scores of 60 out of 100. When the modified Angoff method was applied to the 74th national test of the Korean medical licensing exam, the reference score was 61.4, and when the modified Angoff method was applied to the 81st national test, the reference score was 60.93 points and cut scores of 72.36 and 73.01 points were derived under 3 different conditions [13,14]. When the modified Angoff method was applied to the national examination of medical recorders and radiologists, cut scores of 62.95 points for medical recorders and 71.27 points were obtained for radiologists [14,15].…”

“…Buckendahl et al [17] in 2010 studied the application of the Angoff method for a partial set of items from the Canadian dental licensing test. In previous studies, the panel members reported that it was burdensome to evaluate items outside of one's specialization [13,14]. Dividing the subjects into sub-specializations, such as on the KNLE, and then evaluating and combining each subject better reflected the panel's expertise.…”

Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee

“…In response to this, the Korea Health Personnel Licensing Examination Institute conducted a basic research study to examine possible standard-setting methods [1], and in a recent study, the modified Angoff method was found to be appropriate [2]. National examinations may differ depending on the environment.…”