Does Maximizing Information at the Cut Score Always Maximize Classification Accuracy and Consistency?

Abstract

A common suggestion made in the psychometric literature for fixed-length classification tests is that one should design tests so that they have maximum information at the cut score. Designing tests in this way is believed to maximize the classification accuracy and consistency of the assessment. This article uses simulated examples to illustrate that one can obtain higher classification accuracy and consistency by designing tests that have maximum test information at locations other than at the cut score. We show that the location where one should maximize the test information is dependent on the length of the test, the mean of the ability distribution in comparison to the cut score, and, to a lesser degree, whether or not one wants to optimize classification accuracy or consistency. Analyses also suggested that the differences in classification performance between designing tests optimally versus maximizing information at the cut score tended to be greatest when tests were short and the mean of ability distribution was further away from the cut score. Larger differences were also found in the simulated examples that used the 3PL model compared to the examples that used the Rasch model.An important function of many educational and psychological tests is to classify examinees into different performance categories. In K-12 settings, examinees might be classified into multiple performance categories, such as advanced, proficient, partially proficient, and basic, or simply proficient and not proficient. In licensure and certification (credentialing) testing, examinees are classified as having passed or failed an exam. Key considerations in these contexts are the classification accuracy and consistency of the assessment. Classification accuracy reports the extent to which observed classification agrees with "true" classification, and classification consistency is the proportion of examinees that would be classified into the same performance category over parallel replications of the assessment (Lee, 2010).Measurement experts have expressed a simple, intuitive, and widely accepted approach for how to develop fixed-length tests for maximizing classification accuracy and consistency of exam scores when using item response theory (IRT) models. That approach is to maximize test information near the cut score used to classify examinees (

The Invariance Paradox: Using Optimal Test Design to Minimize Bias

“…The findings of this study are aligned with what one would expect based upon previous empirical work. As Wyse and Babcock () demonstrated, group ability and the location of maximum information of test items have an impact on classification accuracy. It would therefore be expected that classification accuracy would be different for subgroups on an examination, depending upon the location of maximum test information and subgroup means.…”

“…These test lengths were chosen to approximate reliability values of .70, .80, and .90, with .70 indicative of low reliability, .80 indicative of moderate reliability, and .90 indicative of high reliability. Additionally, Wyse and Babcock () found larger differences in classification accuracy when the number of items was at or below 50 items. Reliability was calculated as Rasch person separation reliability (Linacre, ).…”

“…The location of maximum information was manipulated by varying the mean Rasch difficulty of the items. Wyse and Babcock () found that classification accuracy may be highest by maximizing information away from the cut score, depending on examinee ability and reliability. Therefore, the location of maximum information was varied by setting the mean difficulty of items at the cut score, +.5 standard deviations from the cut score, and +1.0 standard deviations from the cut score (i.e., half standard deviation increments from the cut score on the person scale).…”

“…Although maximizing information around the cut score is standard for test construction, there are scenarios where accuracy might be higher by maximizing test information in locations on the ability spectrum other than the cut score. Wyse and Babcock () demonstrated that classification accuracy may be higher by targeting information closer to the group mean when the cut score is far from the group mean and test lengths are shorter (i.e., less reliable scores). For tests with less reliable scores, for example, classification accuracy is higher when information is targeted closer to the group mean, rather than the cut score.…”

Quality of Item Pool (QIP) Index: A Novel Approach to Evaluating CAT Item Pool Adequacy

“…Depending on the purpose of the test, a better item might be the one that maximizes the information at another ability level. For example, in a licensure examination, where the aim is to determine whether examinees are above or below a cut score, test developer might want to administer items that have maximum information at the cut score or somewhere close to cut score (Wyse & Babcock, 2016). The exact locations of the examinees on the ability scale might not be the primary purpose of the examination.…”