A Bayesian Random Effects Model for Testlets

Abstract

Standard item response theory (IRT) models fit to dichotomous examination responses ignore the fact that sets of items (testlets) often come from a single common stimuli (e.g. a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences such as prematurely ending an examination in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g an adaptive test). To model examinations that may be a mixture of independent items and testlets, we modified one standard IRT model to include an additional random effect for items nested within the same testlet. We use a Bayesian framework to facilitate posterior inference via a Data Augmented Gibbs Sampler (DAGS, Tanner and Wong 1987). The modified and standard IRT models are both applied to a data set from a disclosed form of the SAT. We also provide simulation results that indicates that the degree of precision bias is a function of the variability of the testlet effects, as well as the testlet design. Rubin for their helpful suggestions and comments, Ida Lawrence and Miriam Feigenbaum for providing us with the SAT data analyzed in Section 6, and to the two anonymous referees for their careful reading and thoughtful suggestions on an earlier draft. We are also grateful to the Educational Testing Service for providing the resources to do this research.1

Testing for Local Dependence in Rasch’s Multiplicative Gamma Model for Speed Tests

“…In applications, we may encounter tests where some form of (item) clustering is present. If this is the case, LI is not likely to hold (Bradlow, Wainer, & Wang, 1999;Ip, 2001).…”

Bayesian Network Models for Local Dependence Among Observable Outcome Variables

“…The second is to use the superobservable approach of Wainer and Kiely (1987). The third is to introduce additional latent variables to model a testlet effect (Bradlow, Wainer, & Wang, 1999;Wainer et al, 2007). Taking Bayesian networks as our model for the assessment opens the door for new models for conditional dependence.…”

“…It would be reasonable to conclude that the model is underspecified and that tighter priors or other assumptions about the nature of the context effect might be necessary to get the model to converge. The hierarchical models used by Bradlow et al (1999) are one approach.…”

An Assessment of Basic Computer Proficiency Among Active Internet Users: Test Construction, Calibration, Antecedents and Consequences

“…To assess the reasonableness of this assumption, we fit the so-called Testlet IRT model initially proposed by Bradlow, Wainer, and Wang (1999). The Testlet IRT model extends the basic Birnbaum (1968) model to allow for increased dependence of items within the same subscale (in our case computer proficiency domain), by modeling where γ id(j) is an effect for person i on the testlet containing item j, d( j).…”