Optimal Test Assembly in Practice

Abstract

Several techniques have been developed in recent years to generate optimal large-scale assessments (LSAs) of student achievement. These techniques often represent a blend of procedures from such diverse fields as experimental design, combinatorial optimization, particle physics, or neural networks. However, despite the theoretical advances in the field, there still exists a surprising scarcity of well-documented test designs in which all factors that have guided design decisions are explicitly and clearly communicated. This paper therefore has two goals. First, a brief summary of relevant key terms, as well as experimental designs and automated test assembly routines in LSA, is given. Second, conceptual and methodological steps in designing the assessment of the Austrian educational standards in mathematics are described in detail. The test design was generated using a two-step procedure, starting at the item block level and continuing at the item level. Initially, a partially balanced incomplete item block design was generated using simulated annealing, whereas in a second step, items were assigned to the item blocks using mixed-integer linear optimization in combination with a shadow-test approach.

Ant Colony Optimization for Parallel Test Assembly

“…Multiple parallel short tests that can be used interchangeably are needed if item exposure is a concern or repeated testing is envisioned. Usually, parallel short scales are compiled from a larger pilot tested item pool, making it a task of item sampling targeting predefined goals while considering constraints, such as reliability, validity, test fairness, construct coverage or testing time (Kuhn & Kiefer, 2013;Schroeders et al, 2016a;Spaccapanico Proietti et al, 2020;Steger, Jankowsky, et al, 2022;Steger, Weiss, et al, 2022;van der Linden & Glas, 2000;Yan et al, 2014). Different methods have been proposed for this purpose, such as mathematical programming solvers (e.g., Ali & van Rijn, 2016;Becker et al, 2021), machine learning (e.g., Sun et al, 2022), and metaheuristic algorithms (e.g., Leite et al, 2008;Schroeders et al, 2016a).…”

Shortening the Edinburgh postnatal depression scale using optimal test assembly methods: Development of the EPDS‐Dep‐5

“…• The optimal test assembly procedure is datadriven and should be replicated. used in the development of high-stakes educational tests, 26 OTA is being increasingly used to develop shortened versions of patient-reported outcome measures. [27][28][29] This procedure was also shown to be replicable, reproducible, and to produce shortened forms of minimal length compared to alternative methods.…”

“…Optimal test assembly (OTA) is a mixed‐integer programming procedure that uses an estimated item response theory (IRT) model to select the subset of items that maximizes performance with respect to a given metric while satisfying pre‐specified constraints 25 . While more commonly used in the development of high‐stakes educational tests, 26 OTA is being increasingly used to develop shortened versions of patient‐reported outcome measures 27–29 . This procedure was also shown to be replicable, reproducible, and to produce shortened forms of minimal length compared to alternative methods 30 …”

Methods for shortening patient-reported outcome measures

“…3.1.1 Generating candidate forms with optimal test assembly OTA describes an application of mixed integer programming that is used frequently for item selection in designing high-stakes educational tests. 18,19 In OTA, an instrument is constructed from an item pool that is optimal with respect to a chosen objective function, subject to a set of constraints.…”

“…18 OTA is used frequently for item selection in designing high-stakes educational tests. 19 OTA has recently been applied to shorten a 16-item hand function scale in patients with systemic sclerosis to a 6-item shortened form 20 ; however, the details of this procedure have not been rigorously studied.…”