If at First You Don't Succeed Setting Passing Scores When More Than One Attempt Is Permitted

Abstract

For licensing and certification examinations designed to protect the public, passing incompetent candidates (false positives) is a more serious error than failing competent ones. Yet to avoid classification mistakes against individuals and subsequent litigation, current practices for these tests increase the relative risk of false positives. These practices are identified, and one that allows positive errors of measurement arising from multiple opportunities to pass the test is discussed in detail. Unsatisfactory ways to mitigate this problem are considered. Increasing the amount of testing or, if that is not feasible, raising the required passing score for repeaters is recommended.

A Tale of Two Tests (and of Two Examinees)

“…One approach, used by the military, of having different cut scores for each retaking of the exam, while useful in certain circumstances, is difficult to implement because a great deal of empirical information is required to know how the cut score should be shifted on each retaking of the exam. Instead we advocate adopting the averaging approach set forth by Millman (), in which each retaking of the exam is considered as part of one large exam score in which all parts have an equal role; in a classification situation where protecting the public is paramount this approach will likely be politically feasible. This yields a combined score of much greater reliability and hence a much reduced likelihood of a classification error.…”

“…By allowing examinees to retake exams, such an approach can estimate their true scores more accurately, capitalize on the additional data collected during the repeat attempt, and turn a bug into a feature. The suggestion to adapt this approach to scoring repeated attempts on tests was one of several made by Jason Millman (). The details of this are laid out in Section “Rescoring Policies.”…”

“…This notion of reducing variability through averaging is fundamental to the Asian Option (described previously) and was recommended for this application by Jason Millman (). Using a score averaging approach reduces errors in classification.…”

“…This may not be politically feasible for the testing organization, or may require additional changes like adjusting the cut score or weighting more recent attempts more heavily. Other approaches to increase precision may be more politically tenable (see Millman, ) given the potential costs to examinees.…”

“…Several key measurement issues are highlighted by the noncompensatory multiple measure model; classification error (and related measurement error), fairness to examinees (that their classification represents their ability), and the imperative in certification examinations to protect the public from incompetent professionals. Millman () has addressed the related issue of repeated test administrations in the context of licensure and certification and the resulting issue of false positive classification of examinees. One of his suggestions about reducing classification errors is taken up later; however, his findings highlight the important balance between protecting the public by not licensing incompetent individuals (minimizing false positive errors) and not hindering the hiring of competent individuals (minimizing false negative errors).…”