A standard setting method with the best performing students as point of reference: Practical and affordable

Abstract

A standard setting method with the best performing students as point of reference Schotanus, Janke; van der Vleuten, Cees P. M. Take-down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum. AbstractBackground: Teachers involved in test development usually prefer criterion-referenced standard setting methods using panels. Since expert panels are costly, standards are often set by a pre-fixed percentage of questions answered correctly or normreferenced methods aimed at ranking examinees. Aim: To discuss the (dis)advantages of commonly used criterion and norm-referenced methods and present a new compromise method: standards based on a fixed cut-off score using the best scoring students as reference point. Methods: Historical data from 54 Maastricht (norm-referenced) and 52 Groningen (criterion-referenced) tests were used to demonstrate huge discrepancies and variability in cut-off scores and failure rates. Subsequently, the compromise model -known as Cohen's method -was applied to the Groningen tests.Results: The Maastricht norm-referenced method led to a large variation in required cut-off scores (15-46%), but a stable failure rate (about 17%). The Groningen method with a conventional, pre-fixed standard of 60% led to a large variation in failure rates (17-97%). The compromise method reduced variation in required cut-off scores as well as failure rates. Conclusion: Both the criterion and norm-referenced standards, used in practice, have disadvantages. The proposed compromise model reduces the disadvantages of both methods and is considered more acceptable. Last but not least, compared to standard setting methods using panels, this method is affordable.

The Application of Simple, Practical and Affordable Standard Setting Methods in Small Groups of Students

“…Both Cohen-Schotanus and Van der Vleuten (2010) and Taylor (2011) regard the Angoff panel method as the best available, because the panel does examine all questions and attempts to decide how a minimally acceptable candidate could be expected to answer them. However it has been shown experimentally that five different Angoff panels produced quite different pass marks when standard setting the same examination (Boursicot et al 2006), showing that the panel procedure is quite fallible.…”

“…Fixed pass marks and norm referenced methods have often been used to set pass marks (Cohen-Schotanus and Van der Vleuten, 2010). The use of a fixed pass mark produces large variations in failure rates when the difficulty of examinations inevitably varies.…”

“…The pass mark is fixed as a pre-set percentage of this reference point mark. In order to avoid the occasional off scale genius distorting the pass mark Cohen-Schotanus and Van der Vleuten (2010) use the 95 th percentile student mark as the reference point, setting the pass mark as 60% of this, after allowing for randomly obtained marks. This can be expressed by the formula:…”

Setting standards in knowledge assessments: Comparing Ebel and Cohen via Rasch

“…More recently, the 'Cohen' method of standard setting has been advocated (CohenSchotanus and van der Vleuten, 2010;Taylor, 2011). This is seen as a pragmatic method to be employed 'in-house' (i.e.…”

Food for Thought: A New Compromised Method of Standard Setting

“…Other relative methods of standard setting, for example Cohen (Cohen-Schotanus & van der Vleuten, 2010) and Wijnen (van der Vleuten, Verwijnen, & Wijnen, 1996) have great place of use in admission examinations but have major drawbacks when used for progression tests.…”