Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

Abstract

Purpose: Setting standards is critical in health professions. However, appropriate standard setting methods do not always apply to the set cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed from the norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.Methods: This was an explorative study to model of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at seven stations with encountering standardized patients (SPs) and one station with performing skills on a manikin on 15 July 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.Results: The cut score of the norm-referenced method was lower than that of the BGM (p<0.01) and BRM (p<0.02). There was no significant difference in the cut score between the BGM and BRM (p=0.40). The station with the highest standard deviation and the highest proportion of the borderline group showed the largest cut score difference in standard setting methods.Conclusion: Prefixed cut scores by the norm-referenced method without considering station contents or examinee performance can vary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensus on the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study

“…Modifying the cutoff score with the SE of measurement (SEM) is another way to correct the error. The SE focuses on the panel’s decision, whereas the SEM emphasizes the quality of the test and its reliability [ 2 , 4 ]. The calculation formulas are as follows:…”

“…Modifying the cut score with the SE of measurement (SEM) is another way to correct the error. The SE focuses on the panel's decision, whereas the SEM emphasizes the quality of the test and its reliability [2,4]. The calculation formulas are as follows: 𝑆𝐸 = 𝑠 𝑥 /√𝑛 𝑠 𝑥 : SD of observed score x; n: number of samples 𝑆𝐸𝑀 = 𝑆𝐷√1 − 𝑟𝑥𝑥 SD: the SD of test scores; rxx: the reliability of the test…”

“…The standard-setting method has been introduced not only for nationwide high-stakes examinations [ 1 ], but also for single school-based examinations [ 2 ] in Korea. The percent Angoff (PA) method (modified Angoff method) is the oldest and most popular standard-setting method.…”

Newly appointed medical faculty members’ self-evaluation of their educational roles at the Catholic University of Korea College of Medicine in 2020 and 2021: a cross-sectional survey-based study

“…The main changes in teaching methods have included problembased learning aimed at learners' problem-solving skills, judgment, and analytical skills [2,3], team-based learning, which emphasizes cooperative learning, and flipped learning, which allows students to learn basic knowledge before class [4,5]. Changes in educational evaluation have included performance evaluations such as objective structured clinical examinations [6,7,8] and clinical performance examinations [9], which actually evaluate students' performance in clinical situations, criterion-referenced assessments, which evaluate whether individual students have reached their set outcomes, formative assessments aimed at providing feedback to confirm students' level of improvement, and progress tests, which check the degree to which students improve as they progress through the program [10,11]. The changes in medical education as described above have led to more demands than ever to change the role of the teacher in medical education [12].…”