Application of a General Polytomous Testlet Model to the Reading Section of a Large‐scale English Language Assessment

Abstract

Many standardized educational tests include groups of items based on a common stimulus, known as testlets. Standard unidimensional item response theory (IRT) models are commonly used to model examinees' responses to testlet items. However, it is known that local dependence among testlet items can lead to biased item parameter estimates when using standard IRT models, and to overestimated reliability. In this study, a general polytomous testlet model was proposed to account for local dependence in testlet-based tests that contain both dichotomously and polytomously scored items. The proposed model and a standard IRT model were fit to simulated data and several real data sets from the reading sections of a large-scale Englishlanguage test, and model fit was evaluated. Item parameters and test information obtained from the two models were compared to check the impact of local item dependence. In addition, a multidimensional IRT model with simple structure was also fit to the real data sets. Results based on both simulated and real data suggested that local dependence had a small impact on item parameter estimates and a relatively larger impact on test information and reliability. It was also found that the multidimensional IRT model with simple structure fit the real data sets better than the general polytomous testlet model and the standard IRT model did.

Examination of Different Item Response Theory Models on Tests Composed of Testlets

“…In the whole calibration process, the overall and specific dimensions in bi-factor and the general dimension in UIRT and TRT are limited to have the standard normal distribution N (0,1). Thus, parameter estimates obtained from different IRT models are ensured to be at the same scale level (Li Y., Li S., & Wang, 2010).…”

A Short Note on Estimating the Testlet Model With Different Estimators in Mplus

“…The testlet model (e.g., Bradlow, Wang, & Wainer, 1999) is a popular approach used by item response theory (IRT) researchers and practitioners to address the issue of local item dependence (LID) caused by a cluster of items sharing the same stimulus, which can be a reading comprehension passage, a scenario, a graph, and so on. Estimation of this model has usually been conducted with full-information methods, including Markov chain Monte Carlo (MCMC) algorithm (e.g., Koziol, 2016;Li, Bolt, & Fu, 2006) and marginal maximum likelihood estimation (MMLE; e.g., Jiao, Wang, & He, 2013;Li, Li, & Wang, 2010), although Bolt (2005) showed that both full-and limited-information methods can be applied to multidimensional item response theory (MIRT; Reckase, 2009) models. As different estimation methods are implemented in different software programs, researchers and practitioners may have to learn to use a software program with which they might not be familiar in order to use a particular estimation method.…”

Multidimensional item response theory observed score equating methods for mixed-format tests

“…However, in cases where test items are believed to measure multiple abilities, attempts need to be made to model the interaction of test items and examinees as accurately as possible. It is likely that MIRT models provide a better fit than UIRT models in some instances (Li, Li, & Wang, 2010), and as a result, MIRT scale linking and equating procedures are becoming more prevalent. The next section provides background information concerning the advancements that have been made in the area of MIRT equating.…”