Item Position Effects Are Moderated by Changes in Test-Taking Effort

Abstract

This article examines the interdependency of two context effects that are known to occur regularly in large-scale assessments: item position effects and effects of test-taking effort on the probability of correctly answering an item. A microlongitudinal design was used to measure test-taking effort over the course of a large-scale assessment of 60 min. Two components of test-taking effort were investigated: initial effort and change in effort. Both components of test-taking effort significantly affected the probability to solve an item. In addition, it was found that participants' current test-taking effort diminished considerably across the course of the test. Furthermore, a substantial linear position effect was found, which indicated that item difficulty increased during the test. This position effect varied considerably across persons. Concerning the interplay of position effects and test-taking effort, it was found that only the change in effort moderates the position effect and that persons differ with respect to this moderation effect. The consequences of these results concerning the reliability and validity of large-scale assessments are discussed.

A structural equation modeling approach for examining position effects in large-scale assessments

“…The chosen values for the linear position effect were similar to those found in the empirical study as well as the position effects reported in earlier studies (e.g., Debeer and Janssen 2013). Similarly, sample size values resemble the number of examinees from previous empirical studies on item position effects (e.g., Bulut et al 2016;Debeer and Janssen 2013;Qian 2014;Weirich et al 2016). For each crossed condition, 1000 data sets were generated.…”

“…Item position effect, which often is viewed as a context effect in assessments (Brennan 1992;Weirich et al 2016), occurs when the difficulty or discrimination level of a test item varies depending on the location of the item on the test form. For example, the difficulty of an item can increase in later positions due to a fatigue effect or decreasing test-taking effort (Hohensinn et al 2011;Weirich et al 2016).…”

“…For example, the difficulty of an item can increase in later positions due to a fatigue effect or decreasing test-taking effort (Hohensinn et al 2011;Weirich et al 2016). To investigate item position effects, researchers have proposed different approaches using logistic regression (e.g., Davey and Lee 2011;Pomplun and Ritchie 2004), multilevel IRT models based on the GLMM framework (e.g., Albano 2013; Li et al 2012;Weirich et al 2014), and test equating (e.g., Pommerich and Harris 2003; Meyers et al 2009;Store 2013).…”

“…Fourth, once significant item position effects are detected, other manifest and/or latent variables (e.g., gender, test motivation, and test anxiety) can be incorporated into the SEM model to explain the underlying reasons of the found effects. For example, Weirich et al (2016) recently found in an empirical study that item position effects in a large-scale assessment were affected by the examinees' test-taking efforts. Response time effort (Wise and Kong 2005) and disability status (Abedi et al 2007;Bulut et al 2016) are other important factors highlighted in the literature.…”

“…Several methodological studies have focused on item position effects in national and international large-scale assessments, such as the Trends in International Mathematics and Science Study (Martin et al 2004), a mathematics assessment of the Germany Educational Standards (Robitzsch 2009), a mathematical competence test of the Austrian Educational Standards (Hohensinn et al 2011), the Graduate Record Examination (Albano 2013), the Program for International Student Assessment (Debeer and Janssen 2013;Hartig and Buchholz 2012), and a German nationwide large-scale assessment in mathematics and science (Weirich et al 2016). In these studies, item position effects were estimated as either fixed or random effects using multilevel item response theory (IRT) models within the generalized linear mixed modeling framework.…”