Automated Test Assembly in R: The eatATA Package

Abstract

Combining items from an item pool into test forms (test assembly) is a frequent task in psychological and educational testing. Although efficient methods for automated test assembly exist, these are often unknown or unavailable to practitioners. In this paper we present the R package eatATA, which allows using several mixed-integer programming solvers for automated test assembly in R. We describe the general functionality and the common work flow of eatATA using a minimal example. We also provide four more elaborate use cases of automated test assembly: (a) The assembly of multiple test forms for a pilot study; (b) the assembly of blocks of items for a multiple matrix booklet design in the context of a large-scale assessment; (c) the assembly of two linear test forms for individual diagnostic purposes; (d) the assembly of multi-stage testing modules for individual diagnostic purposes. All use cases are accompanied with example item pools and commented R code.

Editorial of the Psych Special Issue “Computational Aspects, Statistical Algorithms and Software in Psychometrics”

“…Becker et al [26] introduce the R package eatATA, which allows the usage of several mixed-integer programming solvers for automated test assembly. The general functionality and the common workflow of eatATA are presented using a minimal example and four more complex use cases.…”

IQB Trends in Student Achievement 2018: A Large-Scale Educational Assessment Study in Germany

“…The achievement test data (plausible values, see section 3.3) are already linked to the previous measurement (2012), which means that the achievement test data are comparable between 2012 and 2018. For further details, see also Becker et al (2019).…”

“…All tasks and items went through a rigorous pre-testing process prior to the study and the majority of the items had already been used in the first national assessment in 2012. 61% of the mathematics items and nearly 100% of the science items were already used in 2012 (Becker et al, 2019). New items for the 2018 assessment were developed and pretested using the same quality standards.…”

Controlling the Speededness of Assembled Test Forms: A Generalization to the Three‐Parameter Lognormal Response Time Model

“…Detailed explanations on general concepts can be found in van der Linden (2005), detailed illustrations on how to use MILP for ATA in practice can, for example, be found in Diao and van der Linden (2011) and Becker et al. (2021). The general idea is to translate test specifications into mathematical constraints.…”

“…To lower the learning curve for test designers and practitioners, we have developed the eatATA R package, which provides a more intuitive user interface. Furthermore, extensive resources for how to implement ATA problems in eatATA exist (Becker et al., 2021). eatATA currently provides access to various free and commercial solvers, namely, GLPK, lpSolve, Symphony, and Gurobi.…”