Incorporating randomness in the Fisher information for improving item‐exposure control in CATs

Abstract

The most commonly employed item selection rule in a computerized adaptive test (CAT) is that of selecting the item with the maximum Fisher information for the estimated trait level. This means a highly unbalanced distribution of item-exposure rates, a high overlap rate among examinees and, for item bank management, strong pressure to replace items with a high discrimination parameter in the bank. An alternative for mitigating these problems involves, at the beginning of the test, basing item selection mainly on randomness. As the test progresses, the weight of information in the selection increases. In the present work we study, for two selection rules, the progressive methods (Revuelta & Ponsoda, 1998) and the proportional method (Segall, 2004a), different functions that define the weight of the random component according to the position in the test of the item to be administered. The functions were tested in simulated item banks and in an operative bank. We found that both the progressive and the proportional methods tolerate a high weight of the random component with minimal or zero loss of accuracy, while bank security and maintenance are improved.

An Enhanced Approach to Combine Item Response Theory With Cognitive Diagnosis in Adaptive Testing

“…We address two issues innate in their original method: noncomparability of the two information addends and the arbitrary selection of weight. Notice that even though the PWKL index (Chang & Ying, 1996) is considered in (7) and in all arguments hereafter, the two new methods that will be introduced below allow any form of information criterion, including the mutual information index (Wang, 2013), or likelihood weighted KL index (Barrada, Olea, Ponsoda, & Abad, 2008), or Shannon entropy index (Xu et al, 2003), in forming the aggregate item selection index. The noncomparability appears as a result of integration.…”

Computerized Adaptive Testing with R: Recent Updates of the Package catR

“…For fixed length CATs, Barrada, Olea, Ponsoda, and Abad (2008) proposed the following equation to relate W to the number of item positions in the test (ranging from 1 to Q):…”

“…The proportional method (Barrada et al 2008;Segall 2004). While the rest of the selection methods implemented in catR are deterministic, the proportional method is stochastic.…”

Examining Content Control in Adaptive Tests: Computerized Adaptive Testing vs. Computerized Adaptive Multistage Testing

“…Much research has been conducted on CAT including the proposal of new item selection methods (e.g., Barrada, Olea, Ponsoda, & Abad, 2008;Chang & Ying, 1996), stopping rules (e.g., Choi, Grady, & Dodd, 2010), and exposure control methods (e.g., Leung, Chang, & Hau, 2003;van der Linden & Chang, 2005). Researched areas of ca-MST include proposals for new routing methods (Luetch, 2000;Thissen & Mislevy 2000), test assembly methods (Luetch, 2000;Luecht & Nungester, 1998), and stage and module specifications (Patsula, 1999).…”