Development of a modified Cohen method of standard setting

Abstract

Background: A new 'Cohen' approach to standard setting was recently described where the pass mark is calculated as 60% of the score of the student at the 95th percentile, after correcting for guessing. Aim: This article considers how two potential criticisms of the Cohen method can be addressed and proposes a modified version, with the assumptions tested using local data. Methods: The modified version removes the correction for guessing and uses the score of the 90th, rather than the 95th percentile student as the reference point, based on the cumulative density functions for 32 modules from one medical school; and incorporates an indirect criterion-referenced passing standard by changing the 60% multiplier to the ratio of the cut score to the score of the student at the 90th percentile on exams that have been standard set using modified Angoff. Results: The assumption that the performance of the 90th percentile student is consistent over time holds for multiple choice questions. Applying the modified Cohen method to the 32 modules generally reduced the variation in failure rate across modules, compared to a fixed pass mark of 50%. Conclusion:The results suggest that the modified Cohen method holds much promise as an economical approach to standard setting.

The Application of Simple, Practical and Affordable Standard Setting Methods in Small Groups of Students

“…On the face of it the modified Cohen method has advantages. It produced generally slightly more consistent fail rates than the original Cohen method, (Table 1) and the data presented here agree with those of Taylor (2011) that the 90 th percentile student may be a better reference point than the 95 th percentile (Table 3). However, some faculty find not using a Wright J MedEdPublish https://doi.org/10.15694/mep.2016.000106…”

“…The standard setting methods described by Cohen-Schotanus and Van der Vleuten (2010) and modified by Taylor (2011), which set the pass marks of exams and tests as a fixed proportion of the mark of top performing students, are simple to use and quick to apply. Both the original method and its modification proved to be highly effective in reducing the variability in failure rates when applied to the examinations and tests taken over a four year period in the first two years of the MBBS programme of a new African medical school with cohort sizes ranging from 42 -54 students.…”

“…Both Cohen-Schotanus and Van der Vleuten (2010) and Taylor (2011) regard the Angoff panel method as the best available, because the panel does examine all questions and attempts to decide how a minimally acceptable candidate could be expected to answer them. However it has been shown experimentally that five different Angoff panels produced quite different pass marks when standard setting the same examination (Boursicot et al 2006), showing that the panel procedure is quite fallible.…”

Setting standards in knowledge assessments: Comparing Ebel and Cohen via Rasch

“…More recently, the 'Cohen' method of standard setting has been advocated (CohenSchotanus and van der Vleuten, 2010;Taylor, 2011). This is seen as a pragmatic method to be employed 'in-house' (i.e.…”

“…in individual medical schools) where the top performing candidates (e.g. the 90 th or 95 th percentile) in an examination are used to estimate or benchmark the difficulty of the test, and then the standard is set at a proportion of this benchmark, possibly with an additional correction for guessing depending on the local context (Taylor, 2011). Hence, Cohen is perhaps best described as a modified norm-reference approach with in-built adjustment for test difficulty.…”