A Strategy for Optimizing Item‐Pool Management

Ariel, Adelaide; Linden, Willem J. van der; Veldkamp, Bernard P.

doi:10.1111/j.1745-3984.2006.00006.x

Cited by 10 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We want to conclude the paper by highlighting three important points. First, whereas we have mainly focused on ATA procedures given an existing item pool, ATA methods can already be used in prior stages, for example, in the stage of item writing (Ariel, van der Linden, & Veldkamp, 2006). This has the advantage that a balanced item pool can be obtained even before ATA algorithms for LSA are implemented, thus enlarging the design space and, hence, the feasibility of an optimal solution.…”

Section: Discussionmentioning

confidence: 99%

Optimal Test Assembly in Practice

Kuhn

Kiefer²

2013

Zeitschrift für Psychologie

View full text Add to dashboard Cite

Several techniques have been developed in recent years to generate optimal large-scale assessments (LSAs) of student achievement. These techniques often represent a blend of procedures from such diverse fields as experimental design, combinatorial optimization, particle physics, or neural networks. However, despite the theoretical advances in the field, there still exists a surprising scarcity of well-documented test designs in which all factors that have guided design decisions are explicitly and clearly communicated. This paper therefore has two goals. First, a brief summary of relevant key terms, as well as experimental designs and automated test assembly routines in LSA, is given. Second, conceptual and methodological steps in designing the assessment of the Austrian educational standards in mathematics are described in detail. The test design was generated using a two-step procedure, starting at the item block level and continuing at the item level. Initially, a partially balanced incomplete item block design was generated using simulated annealing, whereas in a second step, items were assigned to the item blocks using mixed-integer linear optimization in combination with a shadow-test approach.

show abstract

Section: Discussionmentioning

confidence: 99%

Optimal Test Assembly in Practice

Kuhn

Kiefer²

2013

Zeitschrift für Psychologie

View full text Add to dashboard Cite

show abstract

“…According to the related previous researches on the CAT, the features of an optimal item pool should include a sufficient number of high quality items that are targeted to the examinee population and CAT specifications (Ariel et al, 2006;Bergstrom & Lunz, 1999;Boyd et al, 2010;He & Reckase, 2013;Parshall et al, 2002;Reckase, 2010;Wise,1997) Besides, it could be concluded according to the related literature that there have been different findings on the item pool size (Chen, Ankenmann, & Spray, 2003;Flaugher, 2000;Stocking, 1994;Urry, 1977). The fact that the item pool is an important component in CAT and that the item pool characteristics and sizes used in the research are different, led the researchers to work on this issue.…”

Section: Introductionmentioning

confidence: 99%

“…Parshall, Spray, Kalohn, and Davey (2002) stated that an item pool should include a sufficient number of high quality items that are targeted to the examinee population. Ariel, van der Linden, and Veldkamp (2006) stated that an optimal item pool should consist of a maximal number of combinations of items that (a) meet all content specifications for the test, and (b) are most informative at a series of ability levels reflecting the shape of the distribution of the ability estimates for a population of examinees. Wise (1997) stated that a high-quality item pool should contain a sufficient number of useful items that allow for efficient, informative testing at important levels of proficiency.…”

Section: Introductionmentioning

confidence: 99%

Effects of Item Pool Characteristics on Ability Estimate and Item Pool Utilization: A Simulation Study

Öztürk¹,

Şahin²

2018

HUJE

View full text Add to dashboard Cite

Forming an item pool for computerized adaptive testing requires a long and demanding process that may be challenging, both in terms of time and cost. Therefore, one may come across such questions as 'How should an optimal item pool be?' and/or 'How many items should exist in an item pool?' Although research with regard to the features to exist in an optimal item pool vary, there has been no consensus reached about how big the item pool size should be. In the current study, the effect of different item pool size and item distribution on ability estimation and item pool utilization was analysed. 36 different item pools were generated through SimulCAT software. Using 1,000 simulees, single session CAT environments were simulated and two different termination rules were used in the study. Findings of the study indicated that as the size of the item pool increased to a specific size, the precision of measurement increased and the number of unused items decreased. By examining the results according to b parameter, it was found that the effect of b parameter distribution over the results decreased.

show abstract

“…Furthermore, test assembly plays a crucial role in ability assessment as it lies at the basis of the entire test production process: from the earlier stages of item creation to the selection of items for building the test forms. In detail, the requirements of the final tests specified in the test assembly model not only determine the structure of the test forms but also define the composition of the item pool (Ariel & van der Linden, 2006), guiding the item writing process.…”

mentioning

confidence: 99%

Chance-Constrained Automated Test Assembly

Proietti¹,

Matteucci²,

Mignani

et al. 2023

Journal of Educational and Behavioral Statistics

Self Cite

View full text Add to dashboard Cite

Classical automated test assembly (ATA) methods assume fixed and known coefficients for the constraints and the objective function. This hypothesis is not true for the estimates of item response theory parameters, which are crucial elements in test assembly classical models. To account for uncertainty in ATA, we propose a chance-constrained version of the maximin ATA model, which allows maximizing the α-quantile of the sampling distribution of the test information function obtained by applying the bootstrap on the item parameter estimation. A heuristic inspired by the simulated annealing optimization technique is implemented to solve the ATA model. The validity of the proposed approach is empirically demonstrated by a simulation study. The applicability is proven by using the real responses to the Trends in International Mathematics and Science Study (TIMSS) 2015 science test.

show abstract

A Strategy for Optimizing Item‐Pool Management

Cited by 10 publications

References 12 publications

Optimal Test Assembly in Practice

Optimal Test Assembly in Practice

Effects of Item Pool Characteristics on Ability Estimate and Item Pool Utilization: A Simulation Study

Chance-Constrained Automated Test Assembly

Contact Info

Product

Resources

About