A Model and Heuristic For Solving Very Large Item Selection Problems

Abstract

A model for solving very large item selection problems is presented. The model builds on previous work in binary programming applied to test construction. Expert test construction practices are applied to situations in which all specifications for item selection cannot necessarily be met. A heuristic for selecting items that satisfy the constraints in the model also is presented. The heuristic is particularly useful for situations in which the size of the test construction problem exceeds the limits of current implementations of linear programming algorithms. A variety of test construction problems involving real test specifications and item data from actual test assemblies were investigated using the model and the heuristic.There has been considerable interest recently in methods of automated item selection in test construction (e.g., Ackerman). These methods, frequently implemented on microcomputers, provide several potential benefits to test specialists, including reducing labor costs and increasing the efficiency of test construction. They can also increase the consistency of successive forms of a test through the development and codification of specifications that are refined and perfected by test experts.Much of the recent interest in automated item selection methods has been stimulated by the development of large-scale computerized item banks (e.g., Boekkooi-Timminga, 1989;Hsu & Sadock, 1985). These item banks often contain very detailed statistical and content-related information about hundreds or, in many cases, thousands of test items. Computerized item banks make it possible for the test specialist to rapidly assemble representative test forms, review the psychometric characteristics and content attributes of those forms, refine the test specifications as needed, and repeat the process until a satisfactory test is assembled.Research in this area also has been stimulated by developments in item response theory (IRT; Lord, 1952Lord, , 1980. Under IRT, the test information function (TIF) is the sum of the independent item information functions for the items in a test (Lord, 1980). The additivity of item information makes it possible to construct tests to target test information functions (TTIFS), as suggested by Birnbaum (1968) andLord (1980). Test construction problems have been formulated as binary programming problems, which offers several advantages. The problems are binary in the sense that an item is or is not included in the test. This is not to be confused with binary response models, in which items are scored correct or incorrect. The methods described in this paper are independent of the way in which the items are scored. Viewing test construction as a binary programming problem allows statistical and nonstatistical test specifications to be expressed in mathematical terms, as constraints on test optimization. The APPLIED PSYCHOLOGICAL MEASUREMENT

Observed-score equating as a test assembly problem

“…Such models have been proposed earlier, for example, to assemble tests to match a target information function (Swanson & Stocking, 1993;Theunissen, 1985;van der Linden & Boekkooi-Timminga, 1989), to assemble sets of parallel test forms (Adema, 1992;Armstrong, Jones & Wu, 1992 ; Boekkooi - Timminga, 1987Timminga, , 1990, to maximize classical test reliability (Adema & van der Linden, 1989;Armstrong, Jones & Wang, 1994), to match tests item by item van der Linden & Boekkooi-Timminga, 1988), or to implement constrained adaptive testing (van der Linden & Reese, in press). In addition, these models allow for all other test specifications typically constraining the selection of items in a testing program.…”

An Investigation of the Simultaneous Moderation of Average Gender and African‐american Score Differences on a Test of Mathematical Reasoning

“…The weighted deviations model and heuristic of Swanson and Stocking (1993) was used to select items for the artificial tests. The WDM is similar to many models in the decision sciences and is used to select items from a pool of items in such a way as to minimize the weighted sum of deviations from constraints reflecting desirable test properties, as established by test specialists.…”

A Method for Severely Constrained Item Selection in Adaptive Testing

“…A New Methodology The foundation of this new methodology for incorporating expert test development practices in the construction of adaptive tests is the application of a weighted deviations model (WDM) and algorithm for item selection (Swanson & Stocking, 1993). This WDM and algorithm were developed in the context of conventional test assembly paradigms that have been proposed in the literature over the last 10 years.…”

“…The weighted deviations algorithm was developed and investigated in many conventional test construction problems using real item pools (Stocking, Swanson, & Pearlman, 1991, 1993 Swanson & Stocking (1993). Swanson & Stocking (1993) (1986, p. 387) suggested that sets of items could be incorporated into a maximum information adaptive testing paradigm by using a set information function, which is the sum of the item information functions for the items comprising that set.…”

“…Swanson & Stocking (1993) (1986, p. 387) suggested that sets of items could be incorporated into a maximum information adaptive testing paradigm by using a set information function, which is the sum of the item information functions for the items comprising that set. This approach is effective if the tests being constructed are made up entirely of item sets and the number of items to be administered from each set is known in advance.…”