Modeling Item Position Effects Using Generalized Linear Mixed Models

Abstract

Item position effects can seriously bias analyses in educational measurement, especially when multiple matrix sampling designs are deployed. In such designs, item position effects may easily occur if not explicitly controlled for. Still, in practice it usually turns out to be rather difficultor even impossible-to completely control for effects due to the position of items. The objectives of this article are to show how item position effects can be modeled using the linear logistic test model with additional error term (LLTM + e) in the framework of generalized linear mixed models (GLMMs), to explore in a simulation study how well the LLTM + e holds the nominal Type I risk threshold, to conduct power analysis for this model, and to examine the sensitivity of the LLTM + e to designs that are not completely balanced concerning item position. Overall, the LLTM + e proved suitable for modeling item position effects when a balanced design is used. With decreasing balance, the model tends to be more conservative in the sense that true item position effects are more unlikely to be detected. Implications for linking and equating procedures which use common items are discussed. Keywords item position effects, generalized linear mixed models, linear logistic test model, balanced incomplete block designIn educational assessment and in large-scale assessments in particular, item response theory (IRT) models are often applied to estimate the difficulty of items as well as the ability of examinees. Linking and equating procedures are used to define common scales, which allow achievement scores of two nonequivalent samples to be compared even if the two samples share only a subset of common items. The idea is to use a subset of so-called anchor items that are common to both samples to define a common scale. The corresponding design is known as the commonitem nonequivalent groups equating design (

A multilevel study of position effects in PISA achievement tests: student- and school-level predictors in the German tracked school system

“…As such, the designs impose limitations on the identification of certain parameters and might provide less accurate results for domains assessed with fewer item clusters. Therefore, we applaud the recent developments undertaken in PISA, which result in minor domains being assessed by a larger number of item clusters, but we also stress the need to evaluate design options that help to better account for the often unwarranted impact of PEs (Weirich et al, 2014). Wu, M. L., Adams, R. J., Wilson, M. R., & Haldane, S. A.…”

“…Of course, large-scale assessments are limited in their possibilities for employing more complex designs using a larger number of booklets. However, assessment designs suitable for large-scale assessments that have a rather limited number of test booklets could be optimized in order to better account for PEs (Weirich, Hecht, & Böhme, 2014). Although balanced with respect to the positions of item clusters, the matrix designs used in PISA were not developed with respect to the assessment of PEs.…”

A structural equation modeling approach for examining position effects in large-scale assessments

“…For example, the difficulty of an item can increase in later positions due to a fatigue effect or decreasing test-taking effort (Hohensinn et al 2011;Weirich et al 2016). To investigate item position effects, researchers have proposed different approaches using logistic regression (e.g., Davey and Lee 2011;Pomplun and Ritchie 2004), multilevel IRT models based on the GLMM framework (e.g., Albano 2013; Li et al 2012;Weirich et al 2014), and test equating (e.g., Pommerich and Harris 2003; Meyers et al 2009;Store 2013). The purpose of the current study was to introduce a factor analytic approach for modeling item position effects using the SEM framework.…”

“…It is typically assumed that item difficulty linearly increases or decreases as the items are administered in later positions. However, changing item positions can also result in changes in item difficulty as a result of the interaction of item positions and the latent trait (e.g., Debeer and Janssen 2013;Weirich et al 2014). Hence, it is important to evaluate both linear position effects and interaction effects when designing a large-scale assessment containing multiple forms with different item orders or with randomized item ordering.…”

“…To date, three different methodological approaches for investigating item position effects have been described: (1) logistic regression models (e.g., Davey and Lee 2011;Pomplun and Ritchie 2004;Qian 2014); (2) multilevel models based on the generalized linear mixed modeling (GLMM) framework (e.g., Albano 2013; Alexandrowicz and Matschinger 2008;Debeer and Janssen 2013;Hartig and Buchholz 2012;Li et al 2012;Weirich et al 2014); and (3) test equating methods (e.g., Kingston and Dorans 1984;Kolen and Harris 1990;Moses et al 2007;Pommerich and Harris 2003;Meyers et al 2009;Store 2013). Although there are some empirical studies that used the factor analytic methods for modeling position effects (e.g., Bulut et al 2016;Schweizer 2012;Schweizer et al 2009), the current study represents the first study that utilized the SEM framework as a methodological approach for examining item position effects.…”