A Stepwise Test Characteristic Curve Method to Detect Item Parameter Drift

Abstract

An important assumption of item response theory is item parameter invariance. Sometimes, however, item parameters are not invariant across different test administrations due to factors other than sampling error; this phenomenon is termed item parameter drift. Several methods have been developed to detect drifted items. However, most of the existing methods were designed to detect drifts in individual items, which may not be adequate for test characteristic curve–based linking or equating. One example is the item response theory–based true score equating, whose goal is to generate a conversion table to relate number‐correct scores on two forms based on their test characteristic curves. This article introduces a stepwise test characteristic curve method to detect item parameter drift iteratively based on test characteristic curves without needing to set any predetermined critical values. Comparisons are made between the proposed method and two existing methods under the three‐parameter logistic item response model through simulation and real data analysis. Results show that the proposed method produces a small difference in test characteristic curves between administrations, an accurate conversion table, and a good classification of drifted and nondrifted items and at the same time keeps a large amount of linking items.

Evaluating Robust Scale Transformation Methods With Multiple Outlying Common Items Under IRT True Score Equating

“…The Elimination method, the DFIT method, or other outlier detection and elimination methods (He et al, 2013; Raju, 1990) generally have at least three steps: (a) scale transformation, (b) outlier detection and exclusion, and (c) scale transformation with the “cleaned” common item set. The stepwise or sequential methods (Guo et al, 2015) have even more steps. Unlike these methods, the robust scale transformation methods are one-step procedures.…”

“…The research literature has shown many efforts in solving the problem with detecting and eliminating of outlying common items before the scale transformation and equating (e.g., DeMars, 2004; Donoghue & Isham, 1998; Guo, Zheng, & Chang, 2015; Holland & Thayer, 1988; Huynh & Meyer, 2010; Raju, 1990; Veerkamp & Glas, 2000). Results from these studies have shown that the detection and elimination of outlying common items surely improved the stability of scale transformation and increased the accuracy of IRT equating.…”

“…Both phenomena make accurate detection of an outlier more challenging. Stepwise or sequential methods have been proposed to detect and eliminate multiple outlying items by flagging and excluding one outlier at a time (e.g., Guo et al, 2015; Hadi & Simonoff, 1993). However, these methods may result in severe content imbalance when multiple items are eliminated from the common item set because each content area typically includes only a small number of items.…”

Online Calibration of Polytomous Items Under the Generalized Partial Credit Model

“…Also along this line are termination rules—termination rules originally developed for multidimensional CAT (Wang, Chang, & Boughton, 2013) may be adapted into online calibration settings. Finally, using online calibration to build vertical scales (Li & Lissitz, 2012; Tong & Kolen, 2007) and detect item parameter drift (Babcock & Albano, 2012; Guo, Zheng, & Chang, 2015) remains promising future directions.…”

Parameter Drift Detection in Multidimensional Computerized Adaptive Testing Based on Informational Distance/Divergence Measures

“…The instance of parameter drift can be determined by comparing the item parameter estimates directly (Bock et al, 1988) or comparing model fit statistics obtained from the different testing events (Glas, 2000). Alternatively, one can compare test characteristic curves across time points (Guo, Zheng, & Chang, 2015; Wollack, Cohen, & Wells, 2003). Statistical measures that are originally developed for analysis of differential item functioning (DIF) also serve as drift detection measures (e.g., Kim & Cohen, 1991; Lord, 1980; Raju, 1988).…”