Uncertainties in the Item Parameter Estimates and Robust Automated Test Assembly

Veldkamp, Bernard P.; Matteucci, Mariagiulia; Jong, Martijn de

doi:10.1177/0146621612469825

Cited by 21 publications

(25 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In previous papers about robust test assembly (De Jong et al, 2009;Veldkamp, 2013;Veldkamp et al, 2013), uncertainty in test assembly was always related to uncertainty in the item parameter estimates. In this article, uncertainty was related to the violation of the assumption of LI, and the presence of testlet effects in TRT models.…”

Section: Resultsmentioning

confidence: 99%

“…A less conservative alternative was proposed in De Jong et al (2009), where only 1 SD was subtracted. Veldkamp et al (2013) studied the De Jong et al (2009) method more into detail and found that subtracting 1 SD for all items might not be realistic and for long tests it might be too conservative. Soyster (1973) based methods assume all the uncertain coefficients parameters have maximum impact on the solution of a 0-1 LP problem, which is usually not the case in practice.…”

Section: Robust Ata With Testletsmentioning

confidence: 99%

See 1 more Smart Citation

Robust Automated Test Assembly for Testlet-Based Tests: An Illustration with Analytical Reasoning Items

Veldkamp

Paap

2017

Front. Educ.

Self Cite

View full text Add to dashboard Cite

In many high-stakes testing programs, testlets are used to increase efficiency. Since responses to items belonging to the same testlet not only depend on the latent ability but also on correct reading, understanding, and interpretation of the stimulus, the assumption of local independence does not hold. Testlet response theory (TRT) models have been developed to deal with this dependency. For both logit and probit testlet models, a random testlet effect is added to the standard logit and probit item response theory (IRT) models. Even though this testlet effect might make the IRT models more realistic, application of these models in practice leads to new questions, for example, in automated test assembly (ATA). In many test assembly models, goals have been formulated for the amount of information the test should provide about the candidates. The amount of Fisher Information is often maximized or it has to meet a prespecified target. Since TRT models have a random testlet effect, Fisher Information contains a random effect as well. The question arises as to how this random effect in ATA should be dealt with. A method based on robust optimization techniques for dealing with uncertainty in test assembly due to random testlet effects is presented. The method is applied in the context of a high-stakes testing program, and the impact of this robust test assembly method is studied. Results are discussed, advantages of the use of robust test assembly are mentioned, and recommendations about the use of the new method are given.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Robust Ata With Testletsmentioning

confidence: 99%

Robust Automated Test Assembly for Testlet-Based Tests: An Illustration with Analytical Reasoning Items

Veldkamp

Paap

2017

Front. Educ.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Often this parallelism is pursued by minimizing the maximum distance with respect to a specific statistical target of the reference test form, using what is called a Minimax MILP model (van der Linden, ). Commonly only the TIF (Swanson & Stocking, ; van der Linden & Adema, ; Veldkamp, Matteucci, & de Jong, ), or on some occasions only the TCC (Armstrong, Jones, & Kunce, ; Armstrong et al., ) are used as statistical targets. However, both statistical targets can be combined in one Minimax model (Ali & van Rijn, ).…”

Section: Discussionmentioning

confidence: 99%

Evaluating Statistical Targets for Assembling Parallel Mixed‐Format Test Forms

Debeer

Ali

Rijn

2017

J Educational Measurement

View full text Add to dashboard Cite

Test assembly is the process of selecting items from an item pool to form one or more new test forms. Often new test forms are constructed to be parallel with an existing (or an ideal) test. Within the context of item response theory, the test information function (TIF) or the test characteristic curve (TCC) are commonly used as statistical targets to obtain this parallelism. In a recent study, Ali and van Rijn proposed combining the TIF and TCC as statistical targets, rather than using only a single statistical target. In this article, we propose two new methods using this combined approach, and compare these methods with single statistical targets for the assembly of mixed‐format tests. In addition, we introduce new criteria to evaluate the parallelism of multiple forms. The results show that single statistical targets can be problematic, while the combined targets perform better, especially in situations with increasing numbers of polytomous items. Implications of using the combined target are discussed.

show abstract

“…But instead of using the infima of these distributions, they subtracted one posterior standard deviation from the estimated Fisher information as a robust alternative. This approach was even studied more into detail by Veldkamp et al (2013), who studied the effects of uncertainties in various item parameters on Fisher information in the assembly of linear test forms.…”

Section: Robust Test Assemblymentioning

confidence: 99%

Robust Computerized Adaptive Testing

Veldkamp

Verschoor

2019

Theoretical and Practical Advances in Computer-Based Educational Measurement

Self Cite

View full text Add to dashboard Cite

In order to optimize measurement precision in computerized adaptive testing (CAT), items are often selected based on the amount of information they provide about a candidate. The amount of information is calculated using item-and person parameters that have been estimated. Usually, uncertainty in these estimates is not taken into account in the item selection process. Maximizing Fisher information, for example, tends to favor items with positive estimation errors in the discrimination parameter and negative estimation errors in the guessing parameter. This is also referred to as capitalization on chance in adaptive testing. Not taking the uncertainty into account might be a serious threat to both the validity and viability of computerized adaptive testing. Previous research on linear test forms showed quite an effect on the precision of the resulting ability estimates. In this chapter, robust test assembly is presented as an alternative method that accounts for uncertainty in the item parameters in CAT assembly. In a simulation study, the effects of robust test assembly are shown. The impact turned out to be smaller than expected. Some theoretical considerations are shared. Finally, the implications are discussed.

show abstract

Uncertainties in the Item Parameter Estimates and Robust Automated Test Assembly

Cited by 21 publications

References 34 publications

Robust Automated Test Assembly for Testlet-Based Tests: An Illustration with Analytical Reasoning Items

Robust Automated Test Assembly for Testlet-Based Tests: An Illustration with Analytical Reasoning Items

Evaluating Statistical Targets for Assembling Parallel Mixed‐Format Test Forms

Robust Computerized Adaptive Testing

Contact Info

Product

Resources

About