Standard-setting plans for the NBME comprehensive Part I and Part II examinations

Abstract

The National Board of Medical Examiners (NBME) has reviewed its procedure for setting pass-fail standards in conjunction with the introduction of its comprehensive Part I and Part II examinations in 1991. This report gives background information on the procedures used for the past decade to set pass-fail standards for the Part I and Part II examinations, an overview of the NBME's research on standard setting, under way since 1987, and a statement of its plans for determining pass-fail standards for these examinations. In 1981 the NBME changed from the norm-referenced standard, used since the 1950s, to a criterion-group approach to setting pass-fail standards. Although the criterion-group system resulted in more stable standards, it still meant that the standard moved whenever the performance of the reference group changed. After conducting research, surveying constituencies, and examining alternatives, the NBME has adopted a new standard-setting plan that has the following components: a content-based standard-setting procedure; determination of standards by an appropriate group; use of a fixed standard; and periodic review of standards and standard-setting procedures. This new process will produce three types of improvements: it will incorporate deliberations informed by a wide range of information, including content review; annual review of examinees' performances and pass-fail results and triennial restudy of the process will add further quality control; and a fixed standard will mean that comparable performances will be required across administrations in order to pass.

Simulation-Based Mastery Learning Improves Central Line Maintenance Skills of ICU Nurses

“…The 8 nurse educators formed an expert panel, reviewed the data, and used the Angoff and Hofstee standard setting methods to set the final MPS. The Angoff method is used by the National Board of Medical Examiners, 28 whereas the Hofstee method is often preferred for clinical skills examinations. 29 Both approaches use a panel of expert judges composed of individuals with expertise, experience, and knowledge of the subject area.…”

AMEE Guide No. 18: Standard setting in student assessment

“…The criterionreferenced standard is a ® xed standard that may undergo periodic re-evaluation in view of shifts or trends in candidates' performance over time (Nungester et al, 1991). Credentialling organizations may use norm-referenced orientation, in which the standard is based on performance of an external large representative sam ple (norm group) equivalent to th e candidates taking the test.…”

Establishing cutoff scores on assessments of surgical skills to determine surgical competence

“…In addition, the Angoff method was the first of the absolute methods and thus has the longest history of successful use, even in high-stakes testing situations such as the United States Medical Licensing Examination. 3 The Modified Angoff method allows the experts to have actual performance data from a scale as an additional source of information to make a decision. Most standard setting in medicine uses the modified version of the Angoff method, because it is a reasonable, practical, and defensible approach to standard setting.…”

“…Moreover, norm referencing encourages competition, not cooperation, and it is somewhat unstable, as it will shift according to the performance of the norm group. 3 Criterion referencing provides a clear definition of what the trainee should be able to do and it provides a given standard to indicate a competent level of performance. Criterion referencing is also more responsive to the subject matter being taught and allows the teacher and learner to clearly pinpoint capabilities.…”