A structural equation modeling approach for examining position effects in large-scale assessments

Bulut, Okan; Quo, Qi; Gierl, Mark J.

doi:10.1186/s40536-017-0042-x

Cited by 12 publications

(18 citation statements)

References 41 publications

(67 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Section 2.6 , we introduced the FCS imputation models IF1 and IF2 that used

and

in the imputation, respectively. Previous research indicated that item parameters are affected by position effects [ 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 ]. Hence, in our analysis, the FCS imputation models IF1 and IF2 are separately applied for each test booklet.…”

Section: Pisa 2018 Mathematics Case Study: Methodsmentioning

confidence: 99%

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Robitzsch

2021

EJIHPE

View full text Add to dashboard Cite

Missing item responses are prevalent in educational large-scale assessment studies such as the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians have advocated for a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. In an illustrative simulation study, it is shown that the Mislevy-Wu model provides unbiased model parameters. Moreover, the simulation replicates the finding from various simulation studies from the literature that scoring missing item responses as wrong provides biased estimates if the latent ignorability assumption holds in the data-generating model. However, if missing item responses are generated such that they can only be generated from incorrect item responses, applying an item response model that relies on latent ignorability results in biased estimates. The Mislevy-Wu model guarantees unbiased parameter estimates if the more general Mislevy-Wu model holds in the data-generating model. In addition, this article uses the PISA 2018 mathematics dataset as a case study to investigate the consequences of different missing data treatments on country means and country standard deviations. Obtained country means and country standard deviations can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, in the discussion section, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

show abstract

“…In Section 2.6 , we introduced the FCS imputation models IF1 and IF2 that used

and

Section: Pisa 2018 Mathematics Case Study: Methodsmentioning

confidence: 99%

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Robitzsch

2021

EJIHPE

View full text Add to dashboard Cite

show abstract

“…As the cross‐mode correlation incorporates parts of the instrument administered at different positions, a perfect correlation is not expected, even if construct equivalence is given and the measurement model fits perfectly, until interindividual differences in position effects are accounted for (e.g., Bulut, Quo, & Gierl, ; see panel “position effect‐corrected cross‐mode correlation model” in Figure ). The position effect is conceptualized as a difference in the item functioning of identical items at different positions within an assessment.…”

Section: Equivalence Criteriamentioning

confidence: 99%

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

Kroehne

Buerger

Hahnel

et al. 2019

Educational Measurement

View full text Add to dashboard Cite

For many years, reading comprehension in the Programme for International Student Assessment (PISA) was measured via paper‐based assessment (PBA). In the 2015 cycle, computer‐based assessment (CBA) was introduced, raising the question of whether central equivalence criteria required for a valid interpretation of the results are fulfilled. As an extension of the PISA 2012 main study in Germany, a random subsample of two intact PISA reading clusters, either computerized or paper‐based, was assessed using a random group design with an additional within‐subject variation. The results are in line with the hypothesis of construct equivalence. That is, the latent cross‐mode correlation of PISA reading comprehension was not significantly different from the expected correlation between the two clusters. Significant mode effects on item difficulties were observed for a small number of items only. Interindividual differences found in mode effects were negatively correlated with reading comprehension, but were not predicted by basic computer skills or gender. Further differences between modes were found with respect to the number of missing values.

show abstract

“…Various methods are used in large-scale assessments in education in line with this aim. To make the results of these kinds of assessments more reliable, one of the widely used methods in different positions or locations within the tests (Bulut et al, 2017). Thus, problems such as individuals memorizing items or copying answers of other examinees during the test application can be overcomed (Bulut, 2015).…”

Section: Introductionmentioning

confidence: 99%

An Investigation of Item Position Effects by Means of IRT-Based Differential Item Functioning Methods

Soysal

Koğar

2021

International Journal of Assessment Tools in Education

View full text Add to dashboard Cite

In this study, whether item position effects lead to DIF in the condition where different test booklets are used was investigated. To do this the methods of Lord's chi-square and Raju's unsigned area with the 3PL model under with and without item purification were used. When the performance of the methods was compared, it was revealed that generally, the method of Lord's chi-square identified more items with DIF than did the method of Raju's unsigned area. The differentiation of the booklets with respect to item position resulted in a higher number of items displaying DIF with item purification conditions. Based on the findings of the present study, to avoid the occurrence of DIF due to item position effects, it is recommended to position the same items across different booklets in similar locations when forming different booklets. Item Position EffectsThe interaction between the position of a test item in a test booklet and the performance a test taker displays on the same item is called item position effects -IP effects (Qian, 2014). Kingston and Dorans (1984) stated that, in the most classical way, IP effects may emerge in two conditions; namely, items in a measurement instrument that are positioned towards the end may be found easy by test takers owing to practice or learning effect (a positive IP effect) or they can be found difficult owing to fatigue effect (a negative IP effect). An item displaying IP effects means that the item parameters (e.g., difficulty or discrimination) can vary according to the item's position in the booklet (Weirich et al., 2017). For example, Weirich et al. ( 2017) stated that considering IP effects on item difficulty, an item administered at the end of a test often is more difficult than the same item administered at the beginning of the test (p.115). Similarly, Le (2017) concluded that items tend to be more difficult when placed towards the end of the test. The test-takers in this study may have found the items positioned towards the end difficult owing to their decrease in motivation in the exam. However, whatever the underlying reason is, conditions that occur owing to IP effects negatively impact the validity of the results. Various studies have also indicated that it is important to consider position effect to test the validity of an assessment (Hahne, 2008;Hohensinn et al., 2008;Qian, 2014). Studies in the literature investigated whether creating different test forms, arranging the location of the items in the test, and ordering the items from easy to hard or hard to easy affect the individuals' performance or item parameters. However, the results of the studies that examined this subject are not the same. While some studies have determined that the item position has a role on individuals' performance (

show abstract

A structural equation modeling approach for examining position effects in large-scale assessments

Cited by 12 publications

References 41 publications

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

An Investigation of Item Position Effects by Means of IRT-Based Differential Item Functioning Methods

Contact Info

Product

Resources

About