If at First You Don't Succeed: Setting Passing Scores When More than One Attempt Is Permitted

Millman, Jason

doi:10.2307/1176180

Cited by 9 publications

(27 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This last finding offers further support for the use of scores from the second attempt for decision-making purposes. Researchers have expressed concern over the most effective way to combine scores from multiple test administrations (Millman 1989). Although it is common practice to base passfail decisions only on scores from the second attempt, it has been shown that decision consistency improves by averaging scores from the two attempts (Clauser and Nungester 2001).…”

Section: Discussionmentioning

confidence: 99%

Measurement precision for repeat examinees on a standardized patient examination

Raymond

Swygert

Kahraman

2011

Adv in Health Sci Educ

View full text Add to dashboard Cite

Examinees who initially fail and later repeat an SP-based clinical skills exam typically exhibit large score gains on their second attempt, suggesting the possibility that examinees were not well measured on one of those attempts. This study evaluates score precision for examinees who repeated an SP-based clinical skills test administered as part of the US Medical Licensing Examination sequence. Generalizability theory was used as the basis for computing conditional standard errors of measurement (SEM) for individual examinees. Conditional SEMs were computed for approximately 60,000 single-take examinees and 5,000 repeat examinees who completed the Step 2 Clinical Skills Examination(®) between 2007 and 2009. The study focused exclusively on ratings of communication and interpersonal skills. Conditional SEMs for single-take and repeat examinees were nearly indistinguishable across most of the score scale. US graduates and IMGs were measured with equal levels of precision at all score levels, as were examinees with differing levels of skill speaking English. There was no evidence that examinees with the largest score changes were measured poorly on either their first or second attempt. The large score increases for repeat examinees on this SP-based exam probably cannot be attributed to unexpectedly large errors of measurement.

show abstract

Section: Discussionmentioning

confidence: 99%

Measurement precision for repeat examinees on a standardized patient examination

Raymond

Swygert

Kahraman

2011

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

“…The results prompt one to question whether scores from the second time around should be trusted more or less than scores from the first. Millman (1989) and Clauser and Nungester (2001) voiced concern over this issue and suggested alternative strategies for managing multiple scores for repeat candidates, including raising the cut score for all candidates, requiring a higher cut score for repeat candidates, and averaging or otherwise combining multiple scores for repeat candidates. The last of these alternatives seems most palatable and is consistent with the psychometric principle of combining multiple measurements to reduce uncertainty.…”

Section: Discussionmentioning

confidence: 99%

“…Retesting is a source of concern for high-stakes certification and licensure tests intended to protect the public because the presence of false-positive errors means that unqualified examinees are granted a credential to practice their profession (Clauser & Nungester, 2001;Millman, 1989). It is generally desirable to provide examinees with an opportunity to retest and can even be inspiring when they improve their performance.…”

mentioning

confidence: 99%

The Second Time Around: Accounting for Retest Effects on Oral Examinations

Raymond

Luciw-Dubas

2010

Eval Health Prof

View full text Add to dashboard Cite

Years of research with high-stakes written tests indicates that although repeat examinees typically experience score gains between their first and subsequent attempts, their pass rates remain considerably lower than pass rates for first-time examinees. This outcome is consistent with expectations. Comparable studies of the performance of repeat examinees on oral examinations are lacking. The current research evaluated pass rates for more than 50,000 examinees on written and oral exams administered by six medical specialty boards for several recent years. Pass rates for first-time examinees were similar for both written and oral exams, averaging about 84% across all boards. Pass rates for repeat examinees on written exams were expectedly lower, ranging from 22% to 51%, with an average of 36%. However, pass rates for repeat examinees on oral exams were markedly higher than for written exams, ranging from 53% to 77%, with an average of 65%. Four explanations for the elevated repeat pass rates on oral exams are proposed, including an increase in examinee proficiency, construct-irrelevant variance, measurement error (score unreliability), and memorization of test content. Simulated data are used to demonstrate that roughly one third of the score increase can be explained by measurement error alone. The authors suggest that a substantial portion of the score increase can also likely be attributed to construct-irrelevant variance. Results are discussed in terms of their implications for making pass-fail decisions when retesting is allowed. The article concludes by identifying areas for future research.

show abstract

“…A second line of research focuses on methods for selecting cut scores and minimizing the number of examinees who pass due to measurement error. For example, Millman (1989) described how measurement error and retake policies may allow people whose true ability is below the cut score to pass the exam even though they should not pass, and he offered several suggestions to try and reduce these effects. Likewise, Clauser and Wainer (2016) looked at how different retake policies may impact who passes and fails exams and noted that different results can be obtained under different policies.…”

Section: Review Of Previous Researchmentioning

confidence: 99%

What Do We Mean When We Say “Pass Rate”?

Wyse

Anderson

2019

Educational Measurement

View full text Add to dashboard Cite

Pass rates are key assessment statistics which are calculated for nearly all high‐stakes examinations. In this article, we define the terminal, first attempt, total attempts, and repeat attempts pass rates, and discuss the uses of each statistic. We also explain why in many situations one should expect the terminal pass rate to be the highest, first attempt pass rate to be the second highest, total attempts pass rate to be the third highest, and repeat attempts pass rate to be the lowest when repeat attempts are allowed. Analyses of data from 14 credentialing programs showed that the expected relationship held for 13 out of 14 of the programs. Additional analyses of pass rates for educational programs in radiography in one state showed that the general relationship held at the state level, but only held for 6 out of 34 educational programs. It is suggested that credentialing programs need to clearly state their pass rate definitions and carefully consider how repeat examinees may influence pass rate statistics. It is also suggested that credentialing programs need to think carefully about the meaning and uses of different pass rate statistics when choosing which pass rates to report to stakeholders.

show abstract

If at First You Don't Succeed: Setting Passing Scores When More than One Attempt Is Permitted

Cited by 9 publications

References 0 publications

Measurement precision for repeat examinees on a standardized patient examination

Measurement precision for repeat examinees on a standardized patient examination

The Second Time Around: Accounting for Retest Effects on Oral Examinations

What Do We Mean When We Say “Pass Rate”?

Contact Info

Product

Resources

About