Effect of Item Response Theory (IRT) Model Selection on Testlet‐Based Test Equating

Abstract

The local item independence assumption underlying traditional item response theory (IRT) models is often not met for tests composed of testlets. There are 3 major approaches to addressing this issue: (a) ignore the violation and use a dichotomous IRT model (e.g., the 2‐parameter logistic [2PL] model), (b) combine the interdependent items to form a polytomous item and apply a polytomous IRT model (e.g., the graded response model [GRM]), and (c) apply a model that explicitly takes into account the dependence at the item level (e.g., the testlet response theory [TRT] model). In this study, a simulation was conducted to compare the performance of these 3 approaches on number‐correct score equating when degrees of testlet effect were manipulated. The traditional equipercentile method was used as an evaluation baseline. The results show that the 2PL and the TRT approaches produce comparable results that more closely agree with the results of the equipercentile method than the GRM does. And the number‐correct equating using the 2PL is robust to the violation of local item independence.

An Extension of Testlet-Based Equating to the Polytomous Testlet Response Theory Model

“…In terms of testlet effect, it was seen that both the 2PL model and GRM were more sensitive, whereas the TRT model and GRTM seemed relatively robust as testlet effect increased from low to high. This general pattern has been consistently observed in the previous study with the comparison of different IRT models on testlet-based test equating for the dichotomous items (Cao et al, 2014), but the previous study has not taken the polytomous items into consideration. Concerning the length of testlet items, it was clear, as discussed earlier, that the TRT model and GRTM were more accurate as the length of testlet items increased than were the 2PL model and GRM.…”

“…In the current practice of educational measurement, test equating is a vital step to put scores from different forms onto a same scale. However, in most large-scale testing programs, it is common for a standardized test to consist of testlets ( Bradlow et al, 1999 ; Rijmen, 2009 ; Cao et al, 2014 ; Tao and Cao, 2016 ). A testlet is defined as an aggregation of items which are based on a common stimulus ( Wainer and Kiely, 1987 ; Bradlow et al, 1999 ).…”

“…Responses to items within a testlet often tend to violate the local item independence. For example, some examinees that are more familiar with the background information covered by the testlet may have a higher probability to correctly answer the items of a specific testlet ( Rijmen, 2009 ; Cao et al, 2014 ; Tao and Cao, 2016 ). Although researchers have conducted an abundance of studies to propose different approaches to handle local item dependence (LID), little research in the literature has focused on the performance of different approaches to accommodate LID on testlet-based test equating.…”

“…Studies have shown that the accuracy of parameter estimation produced by the testlet response theory (TRT) model is higher than the traditional item response theory (IRT) model where LID was present ( Bradlow et al, 1999 ; Wainer and Wang, 2000 ; Wainer et al, 2000 ; Zhang, 2010 ; Koziol, 2016 ). However, numbers of studies were based on dichotomous items ( Wainer and Wang, 2000 ; Rijmen, 2009 ; Cao et al, 2014 ). Researchers have found that although the polytomous IRT models suffer the problem of losing response pattern information, they are still much easier in interpretation and implementation ( Sireci et al, 1991 ; Zenisky et al, 2002 ; Cao et al, 2014 ).…”

Monte Carlo Simulation in Item Response Theory Applications Using SAS

“…This finding was not informed by analytical strategies, but instead using a Monte Carlo simulation study (MCSS; also referred to simulation for short). "The [two-parameter logistic IRT] equating method was quite robust to the violation of local item independence " Cao, Lu, and Tao (2014), another finding rooted in MCSS. These are just two examples of the utility of simulation studies.…”