Moderating Possibly Irrelevant Multiple Mean Score Differences on a Test of Mathematical Reasoning

Abstract

A pool of items from operational tests of mathematical reasoning was constructed to investigate the feasibility of using automated test assembly (ATA) methods to simultaneously moderate possibly irrelevant differences between the performance of women and men, and African American and White test takers. None of the artificial tests exhibited substantial impact moderation, although the estimated mean scaled score differences for the relevant population indicated a modest move in the intended direction: the difference between scaled score means was reduced by about 20%for women and men and about 9%for African American and White test takers. Although many issues in the implementation of this methodology remain to be solved, the consideration of impact in ATA, along with the maintenance of the detailed test plan, appears to be a potential method of moderating possibly irrelevant mean test score differences.

An Empirical Investigation of Impact Moderation in Test Construction

“…Stocking et al (1998a), estimated the same result for their moderated impact tests. In addition, a larger average standard error of measurement can be expected for tests designed to have moderated impact when compared to tests assembled without regard to impact.…”

“…The first method, called "test construction" (TC), uses the WDM directly to simultaneously satisfy all statistical and nonstatistical constraints on item selection, including the moderation of the three different kinds of impact of interest. This is the same approach that was used in the previous study by Stocking et al (1998a). Two versions were tried, one in which a small moderation in the three kinds of impact was the goal (TC-S), and a second in which a larger moderation in the three kinds of impact was the goal (TC-L).…”

“…The horizontal line plotted at 1.0 indicates that the two sections are equally efficient. Stocking, et al (1998a) suggested that tests produced by a process that deliberately sought to moderate impact would likely be less efficient than tests assembled without regard to impact at middle levels of ability and more efficient at more extreme (low or high) levels. This suggestion was based on the observation that moderated impact tests tend to have easier and harder items than tests constructed ignoring impact.…”

“…Still other authors argue that the only irrelevant differences are those remaining after conditioning on test score, which has lead to the study of Differential Item Functioning (DIF; Holland and Thayer, 1988). Stocking, Jirele, Lewis, and Swanson (1998a) sought to simultaneously moderate the mean score differences between women and men, and African American test takers and White test takers, without changing the construct measured by a test of mathematical skills. They used methods of automated test assembly (ATA) (see, for example, Luecht, 1998, van der Linden, 1998, and Wightman, 1998 as a tool in the assembly of moderated impact tests by adding the goal of moderating mean group differences to all the other goals that are used in test assembly.…”

The New Faces of Fairness

“…88-91). Similarly, Stocking, Jirele, Lewis, and Swanson (1998) have demonstrated that different sets of items, all of which meet the same detailed set of test specifications, can produce variations in the size of group differences.…”