Computerized Adaptive Testing in Early Education: Exploring the Impact of Item Position Effects on Ability Estimation

Abstract

Studies have shown that item difficulty can vary significantly based on the context of an item within a test form. In particular, item position may be associated with practice and fatigue effects that influence item parameter estimation. The purpose of this research was to examine the relevance of item position specifically for assessments used in early education, an area of testing that has received relatively limited psychometric attention. In an initial study, multilevel item response models fit to data from an early literacy measure revealed statistically significant increases in difficulty for items appearing later in a 20-item form. The estimated linear change in logits for an increase of 1 in position was .024, resulting in a predicted change of .46 logits for a shift from the beginning to the end of the form. A subsequent simulation study examined impacts of item position effects on person ability estimation within computerized adaptive testing. Implications and recommendations for practice are discussed.

Item position effects in listening but not in reading in the European Survey of Language Competences

“…Data. Studies on item position effects in large-scale assessments mostly focused on national assessments (e.g., Albano, Cai, Lease, & McConnel, 2019;Hohensinn, Kubinger, Reif, Schleicher, & Khorramdel, 2011), a national sample of an international assessment (e.g., , or modeled the results of an international assessment for each country separately (e.g., Debeer et al, 2014;Hartig & Buchholz, 2012;Wu et al, 2019). In the present study, we will show that item position effects can also be estimated in a single model in which separate position effects for each region can be estimated.…”

Towards Association Rule-based Item Selection Strategy in Computerized Adaptive Testing

“…Over the years, in different projects, various tools have been applied in the development of the phases that make up the CATs, for example: threeparameter logistic model for item calibration (Lee et al, 2018); maximum likelihood estimation for the evaluator's skill estimation (Albano et al, 2019); and root mean square differences as an evaluation criterion (Stafford et al, 2019), among others. Specifically, for the item selection stage, work has been done to solve the problems presented by Fisher's Maximum Information, using other selection strategies, for example, Bayesian networks (Tokusada and Hirose, 2016), Greedy algorithm (Bengs, Brefeld and Krohne, 2018), Kullback-Leibler Information (Chen et al, 2017), Minimum Expected Subsequent Variance (Rodríguez-Cuadrado et al, 2020), to mention a few which, while they have achieved favorable results, most have only been in studies of simulation and not in real application.…”

“…Computer Adaptive Testing -CAT- (Chen, Chao and Chen, 2019) has revolutionized the traditional way of evaluating, since it dynamically selects and manages the most appropriate questions depending on the previous answers given by the examinees. One of the central components of a CAT is the item selection criterion (Miyazahua and Ueno, 2019), although the most widely used criterion is Fisher's Maximum Information (Albano et al, 2019), it presents several weaknesses that generate a certain degree of mistrust, for example, bias in the item selection, estimation errors at the start of the exam, or the same question being displayed repeatedly to the tested one (Sheng, Bingwei and Jiecheng, 2018;Du, Li and Chang, 2018;Lin and Chang, 2019;Yigit, Sorrel and de la Torre, 2019;Ye and Sun, 2018). Therefore, in this paper the development of a CAT system that uses association rules for the selection of items is proposed, focusing on using the potential advantages of association rules to find relationships between the questions answered correctly or incorrectly and the questions answered correctly, and thus present the most appropriate questions (most likely to answer correctly) in the tests, according to the responses of the evaluated, considering the best rules (stored in the database of students who submitted the same test previously) with greater support and confidence.…”

Contextual Interference Effects in Early Assessment: Evaluating the Psychometric Benefits of Item Interleaving

“…In some cases, effects appear to be negligible (e.g., Li et al, 2012). In others, items have tended to become more difficult when administered at the end of a test compared with the beginning (e.g., Pomplun and Ritchie, 2004;Meyers et al, 2008;Albano, 2013;Debeer and Janssen, 2013;Albano et al, 2019). Studies have also identified items that decrease in difficulty by position (e.g., Kingston and Dorans, 1984).…”

“…Here, it may be that challenging cognitive tasks or novel item types that are initially unfamiliar to test takers become easier as test takers warm up to them. Whatever their direction, non-negligible position effects pose a threat to assumptions of item parameter invariance over forms, and need to be evaluated in programs where item position may vary, for example, with CAT (Albano, 2013;Albano et al, 2019).…”