Automated Test Assembly for Cognitive Diagnosis Models Using a Genetic Algorithm

Abstract

Much recent psychometric literature has focused on cognitive diagnosis models (CDMs), a promising class of instruments used to measure the strengths and weaknesses of examinees. This article introduces a genetic algorithm to perform automated test assembly alongside CDMs. The algorithm is flexible in that it can be applied whether the goal is to minimize the average number of classification errors, minimize the maximum error rate across all attributes being measured, hit a target set of error rates, or optimize any other prescribed objective function. Under multiple simulation conditions, the algorithm compared favorably with a standard method of automated test assembly, successfully finding solutions that were appropriate for each stated goal.

Attribute Discrimination Index-Based Method to Balance Attribute Coverage for Short-Length Cognitive Diagnostic Computerized Adaptive Testing

“…As mentioned in the Introduction, in some situations, even though adequate items are used to measure each attribute, the estimated accuracy may differ across attributes (Finkelman et al, 2009). The number of items measuring each attribute may be the necessary condition to improve the AMPs' accuracy.…”

“…The findings from Cheng (2010) indicated that the correct classification rate had increased when the number of items measuring each attribute is adequate, which implied that there is a positive correlation between the numbers of items measuring each attribute and the correct classification rate. However, Finkelman et al (2009) pointed out that, in some situations, even if the test contained adequate numbers of items to measure each attribute, different measurement accuracy could occur across the attributes. In other words, the number of items measuring each attributes maybe not the essential factor that affects the measurement accuracy of latent skills.…”

Evaluating Statistical Targets for Assembling Parallel Mixed‐Format Test Forms

“…Different methods are available. Some are based on sampling-and-classification methods such as the Cell Only and the Cell and Cube methods (Chen et al, 2012), while others rely on constrained combinatoral optimization techniques (e.g., Finkelman, Kim, and Roussos, 2009;Luecht, 1998;van der Linden, 2005). The most commonly used optimization technique involves translating the ATA problem to a mixed integer linear programming (MILP) model (Diao and van der Linden, 2011;Theunissen, 1985;van der Linden, 2005).…”

Simultaneously construct IRT-based parallel tests based on an adapted CLONALG algorithm

“…The adapted CLONALG is evaluated by comparing it with LP and GA. LP is widely used in the test construction problem and GA has been reported an efficient method for solving a large and complex test construction problem in recent years [11,14,25,30]. The Premium Solver Platform with the Large-Scale LP Solver Engine is used as the LP solver tool [8].…”

“…2 Concatenate multiple strings to regarded as parallel tests [16] Because the test construction is a combinatorial optimization problem the number of combinations grows exponentially with the item bank size. Many researchers have applied novel heuristic algorithms such as the neural network technique [24], the Monte Carlo random search [5], the tabu search algorithm [15], the particle swarm optimization algorithm [32], the immune algorithm [18], and GA [11,14,25] to determine a near-optimal test in recent years. These studies focused on the single test construction.…”