Group Differences in the Value of Subscores: A Fairness Issue

Abstract

The aim of this paper was to study fairness in testing by analyzing the quality of subscores for different groups of test takers. This is done by studying the value added ratio (VAR) for all subscores in the test, which here is a Swedish college admission test. Comparisons were made between test takers who take the regular test and test takers who are taking the test with extended time adaptation, as well as between males and females. Significant group differences in such subscore value would raise questions about fairness for that testing program. In particular, differences in subscore value between subgroups based on accommodations could indicate problems with the appropriateness of the accommodation. We found that the value added ratio (VAR) for all subscores in the studied test were slightly larger for males than for females. There were some differences between subgroups with respect to the extended time accommodation, but they were not systematic in the same way as for males and females. While the differences in the subscores' added value were small in general, they may in some instances be large enough to pose a threat to a valid and fair interpretation and use of test scores, and thus raises questions about fairness.

Multivariate Structural Equation Modeling Techniques for Estimating Reliability, Measurement Error, and Subscale Viability When Using Both Composite and Subscale Scores in Practice

“…A common question addressed when using measures that produce both subscale and composite scores is whether subscale scores provide information or "added value" beyond that provided by composite scores. A useful classical test theory-based method for answering this question originally proposed by Haberman ([64]; also see [65][66][67][68][69]) is to compare proportional reductions in mean squared error (PRMSE) in estimating a subscale's true scores using observed scores from the subscale versus its associated composite. Vispoel, Lee, Hong, and Chen ( [17]; also see [59,70]) noted that Haberman's procedure also can be readily applied to G-theory designs by substituting universe score for true score estimation.…”

“…In essence, a PRMSE index represents an estimate of the proportion of true or universe score variance that is accounted for by targeted observed scores (subscale or composite; [67,68]). Once PRMSEs are derived for a subscale and its associated composite scale, they can be placed in Equation ( 9) to form a value-added ratio (VAR; [69]).…”