Comparisons of Methodologies and Results in Vertical Scaling for Educational Achievement Tests

Ye, Tong; Kolen, Michael J.

doi:10.1080/08957340701301207

Cited by 56 publications

(112 citation statements)

References 21 publications

Supporting

Mentioning

101

Contrasting

Unclassified

Order By: Relevance

“…Currently, a large number of studies (e.g., Hanson & Beguin, 2002;Kang & Petersen, 2009;Kim & Cohen, 2002;Meng, 2007;Tong & Kolen, 2007) have explored factors affecting IRT vertical scaling for cell (1, 1) when both assumptions hold, which is nearly true in reality. For cell (0,1) where tests are multidimensional and construct invariant across grades, a few studies (Beguin & Hanson, 2001;Beguin, Hanson, & Glas, 2000;Patz & Yao, 2007;Simon, 2008) have applied multidimensional IRT models to vertical scaling.…”

Section: Assumption Revisit and Gap In Literaturementioning

confidence: 99%

“…were also explored by some studies (Hendrickson et al, 2004;Hendrickson et al, 2005;Hendrickson, Cao, Chae, & Li, 2006;Meng, Kolen, & Lohman, 2006;Tong & Kolen, 2007). These studies consistently concluded that MLE yielded larger within-grade variability and smaller effect sizes than the Bayesian based methods (e.g., EAP and MAP); but all types of proficiency estimates resulted in very similar proficiency means and mean difference patterns across grades.…”

Section: Other Factorsmentioning

confidence: 99%

“…The effects of data collection design (e.g., common item design and scaling test design) on performance of IRT vertical scaling were investigated by a few studies (Hendrickson, Kolen, & Tong, 2004;Hendrickson, Wei, Kolen, & Tong, 2005;Tong & Kolen, 2007). Findings from these studies consistently revealed that the common item design yielded decreasing variability of latent traits across grades, and larger effect sizes (indicating more growth) compared to those from the scaling test design.…”

Section: Other Factorsmentioning

confidence: 99%

“…For UIRT vertical scaling, sample size is included in examining its effect on the performance of vertical scaling in many studies (Beguin & Hanson, 2001;Beguin et al, 2000;Hanson & Beguin, 2002;Kang & Petersen, 2009;Kim & Cohen, 2002;Lei & Zhao, 2010;Meng, 2007;Paek et al, 2008;Smith et al, 2008;Tong & Kolen, 2007;Yon, 2006).…”

Section: Sample Sizementioning

confidence: 99%

“…Using three-parameter logistic (3PL) IRT models (Lord, 1980), Tong and Kolen (2007) set the sample size at three levels: 500, 2000, and 8000; Hanson and Beguin (2002) set two levels: 1000, and 3000; Kang and Petersen (2009) also set two levels: 500, and 2000. According to a rule of thumb suggested by Harris (1993), approximately 1500 examinees per form were adequate for the 3PL IRT model.…”

Section: Sample Sizementioning

confidence: 99%

See 4 more Smart Citations

Exploring the Full-Information Bifactor Model in Vertical Scaling With Construct Shift

Lissitz

2012

Applied Psychological Measurement

View full text Add to dashboard Cite

To address the lack of attention to construct shift in IRT vertical scaling, a bifactor model is proposed to estimate the common dimension for all grades and the grade-specific dimensions. The bifactor model estimation accuracy is evaluated through a simulation study with manipulated factors of percent of common items, sample size, and degree of construct shift. In addition, the unidimensional IRT (UIRT) estimation model that ignores construct shift is examined to represent the current practice for IRT vertical scaling; comparisons on parameter estimation accuracy of the bifactor and UIRT models are discussed.The major findings of the simulation study are (1) bifactor models are well recovered overall, even though item discrimination parameters are underestimated to a small degree; (2) item discrimination parameter estimates are overestimated in UIRT models due to the effect of construct shift; (3) person parameters of UIRT models are less accurately estimated than that of bifactor models, and the accuracy decreases as the degree of construct shift increases; (4) group mean parameter estimates of UIRT models are less accurate than that of bifactor models, and a large effect due to construct shift is found for the group mean parameter estimates of UIRT models.The real data analysis provides an illustration of how bifactor models can be applied to a problem involving for vertical scaling with construct shift. General procedures for testing practice are also discussed.

show abstract

Section: Assumption Revisit and Gap In Literaturementioning

confidence: 99%

Section: Other Factorsmentioning

confidence: 99%

Section: Other Factorsmentioning

confidence: 99%

Section: Sample Sizementioning

confidence: 99%

Section: Sample Sizementioning

confidence: 99%

See 3 more Smart Citations

Exploring the Full-Information Bifactor Model in Vertical Scaling With Construct Shift

Lissitz

2012

Applied Psychological Measurement

View full text Add to dashboard Cite

show abstract

Effectiveness of Item Response Theory (IRT) Proficiency Estimation Methods Under Adaptive Multistage Testing

Kim

Moses

Yoo

2015

ETS Research Report Series

View full text Add to dashboard Cite

The purpose of this inquiry was to investigate the effectiveness of item response theory (IRT) proficiency estimators in terms of estimation bias and error under multistage testing (MST). We chose a 2‐stage MST design in which 1 adaptation to the examinees' ability levels takes place. It includes 4 modules (1 at Stage 1, 3 at Stage 2) and 3 paths (low, middle, and high). When creating 2‐stage MST panels (i.e., forms), we manipulated 2 assembly conditions in each module, such as difficulty level and module length, to see if any interaction existed between IRT estimation methods and MST panel designs. For each panel, we compared the accuracy of examinees' proficiency levels derived from 7 IRT proficiency estimators. We found that the choice of Bayesian (prior) and non‐Bayesian (no prior) estimators was of more practical significance than the choice of number‐correct versus item‐pattern scoring. For the extreme proficiency levels, the decrease in standard error compensated for the increase in bias in the Bayesian estimates, resulting in smaller total error. Possible score changes caused by the use of different proficiency estimators would be nonnegligible, particularly for the extreme proficiency level examinees. The impact of misrouting at Stage 1 was minimal under the MST design used in this study.

show abstract

Investigating Robustness of Item Response Theory Proficiency Estimators to Atypical Response Behaviors Under Two‐Stage Multistage Testing

Kim

Moses

2016

ETS Research Report Series

View full text Add to dashboard Cite

The purpose of this study is to evaluate the extent to which item response theory (IRT) proficiency estimation methods are robust to the presence of aberrant responses under the GRE® General Test multistage adaptive testing (MST) design. To that end, a wide range of atypical response behaviors affecting as much as 10% of the test items were simulated using a generic GRE 2‐stage MST and the 2‐parameter logistic (2PL) IRT model. As expected, some differences were found among the 5 estimators in terms of the recovery of the true theta ability; for example, Bayesian estimators had lower error variance and their estimates were regressed to the mean. Once the IRT theta estimates were scaled onto a comparable reporting score scale, however, it was found that all the estimation methods investigated, including the one currently used to score GRE MSTs, were equally robust under the simulated conditions.

show abstract

Comparisons of Methodologies and Results in Vertical Scaling for Educational Achievement Tests

Cited by 56 publications

References 21 publications

Exploring the Full-Information Bifactor Model in Vertical Scaling With Construct Shift

Exploring the Full-Information Bifactor Model in Vertical Scaling With Construct Shift

Effectiveness of Item Response Theory (IRT) Proficiency Estimation Methods Under Adaptive Multistage Testing

Investigating Robustness of Item Response Theory Proficiency Estimators to Atypical Response Behaviors Under Two‐Stage Multistage Testing

Contact Info

Product

Resources

About