Standard Setting: An Alternative Approach

Abstract

Since 1971 there have been a number of studies in which a cut score has been set using a method proposed by Angoff (1971). In this method, each member of a panel of judges estimates for each test question the proportion correct for a specific target group of examinees. Prior and contemporary research suggests that this is a difficult task for judges. Angoff also proposed that judges simply indicate whether or not an examinee from the target group will be able to answer each question correctly (the yes/no method). We report on the results of two studies that compare a yes/no estimation with a proportion correct estimation. The two studies demonstrate that both methods produce essentially equal cut scores and that judges find the yes/no method more comfortable to use than the estimated proportion correct method.

Certification in neurodevelopmental disabilities: The development of a new subspecialty and results of the initial examinations

“…Before test administration the committee also participated in a standard setting exercise to set the initial pass/fail level for the examination [Impara et al, 1997]. In 2002, statistical equating procedures were used to establish the standard based on the subset of previously used questions [Wright and Stone, 1979].…”

Application of the Rasch Model and the Bookmark Method in Setting Cut Scores in Mathematics

“…Using IRT model, relationship between student's ability and item's difficulty, the Bookmark method helps to determine an ability level that represent minimal competency in terms of item difficulty. The advantages offered by IRT calibration provide significant advantages compared to estimation by panelists [8], [9]. The Bookmark method typically proceeds in three rounds.…”

Using item response theory to explore the psychometric properties of extended matching questions examination in undergraduate medical education

“…The focus of the analysis is on the total test score; frequency of correct responses (to indicate question difficulty); frequency of responses (to examine distractors); reliability of the test and item-total correlation (to evaluate discrimination at the item level) [8-11]. Although these statistics have been widely used, one limitation is that they relate to the sample under scrutiny and thus all the statistics that describe items and questions are sample dependent [12].…”