Modeling Omitted and Not-Reached Items in IRT Models

Rose, Norman; Davier, Matthias von; Nagengast, Benjamin

doi:10.1007/s11336-016-9544-7

Cited by 62 publications

(88 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Hence, missing responses due to omitted items can be informative and should not be ignored. Yet, treating them as incorrect may also lead to biased person and item parameter estimates (e.g., Rose et al 2017). As PISA did, missing responses due to omitted items were treated as incorrect in the current study.…”

Section: Limitationsmentioning

confidence: 87%

Predictors of individual performance changes related to item positions in PISA assessments

Debeer

Buchholz

et al. 2019

Large-scale Assess Educ

View full text Add to dashboard Cite

Background: Item position effects have been a common concern in large-scale assessments as changing the order of items in booklets may have an undesired effect on test performance. If every test taker would be affected by the effect in the very same way, comparisons between groups of individuals would still be valid. However, research has shown that in addition to a general fixed effect of item positions, the extent of the effect varies considerably across individuals. These individual differences are referred to as persistence. Test takers with a high level of persistence are able to keep up their performance better throughout the test administration, whereas those with a lower level of persistence show a larger decline in their test performance. Methods: The present study applied a multilevel extended item response theory (IRT) framework and used the data from the PISA 2006 science, 2009 reading, and 2012 mathematics assessments. The first objective of this study is to provide a systematic investigation of item position effects across the three PISA domains, partially replicating the previous studies on PISA 2006 and 2009. Second, this study aims to gain a better understanding of the nature of individual differences in position effects by relating them to student characteristics. Gender, socioeconomic status, language spoken at home, and three motivational scales (enjoyment of doing the subject being assessed, effort thermometer, perseverance) were used as person covariates for persistence. Results: This study replicated and extended the results found in previous studies. An overall negative item cluster position effect and significant individual differences in this effect were found in all the countries in the three PISA domains. Furthermore, the most frequently observed effect of person covariates on persistence is gender, with girls keeping up their performance better than boys. Other predictors showed little or inconsistent effects on persistence. Conclusions: Our study demonstrated inter-individual differences as well as group differences in item position effects, which may threaten the comparability between persons and groups. The consequences and implications of item position effects and persistence for the interpretation of PISA results are discussed.

show abstract

Section: Limitationsmentioning

confidence: 87%

Predictors of individual performance changes related to item positions in PISA assessments

Debeer

Buchholz

et al. 2019

Large-scale Assess Educ

View full text Add to dashboard Cite

show abstract

“…Advanced psychometric approaches have been mainly focused on dealing with the fact that missing data due to items not being reached are not random (Mislevy and Wu, 1996 ) but rather are often related to ability (Rose et al, 2010 ; Pohl et al, 2014 ) and therefore non-ignorable. The idea is to model the respondents' tendency to have missing values and its relationship to the measured ability, which can be done either by modeling the latent missingness propensity (Glas and Pimentel, 2008 ; Debeer et al, 2017 ) or by using the number of not reached items in the model either as a continuous covariate (with linear and possible non-linear effects) in a latent regression or as a grouping variable in a multi-group IRT model (Rose et al, 2010 , 2017 ; Rose, 2013 ).…”

Section: Introductionmentioning

confidence: 99%

On the Importance of the Speed-Ability Trade-Off When Dealing With Not Reached Items

Tijmstra

Bolsinova

2018

Front. Psychol.

View full text Add to dashboard Cite

In many applications of high- and low-stakes ability tests, a non-negligible amount of respondents may fail to reach the end of the test within the specified time limit. Since for respondents that ran out of time some item responses will be missing, this raises the question of how to best deal with these missing responses for the purpose of obtaining an optimal assessment of ability. Commonly, researchers consider three general solutions: ignore the missing responses, treat them as being incorrect, or treat the responses as missing but model the missingness mechanism. This paper approaches the issue of dealing with not reached items from a measurement perspective, and considers the question what the operationalization of ability should be in maximum performance tests that work with effective time limits. We argue that the target ability that the test attempts to measure is maximum performance when operating at the test-indicated speed, and that the test instructions should be taken to imply that respondents should operate at this target speed. The phenomenon of the speed-ability trade-off informs us that the ability that is measured by the test will depend on this target speed, as different speed levels will result in different levels of performance on the same set of items. Crucially, since respondents with not reached items worked at a speed level lower than this target speed, the level of ability that they have been able to display on the items that they did reach is higher than the level of ability that they would have displayed if they had worked at the target speed (i.e., higher than their level on the target ability). Thus, statistical methods that attempt to obtain unbiased estimates of the ability as displayed on the items that were reached will result in biased estimates of the target ability. The practical implications are studied in a simulation study where different methods of dealing with not reached items are contrasted, which shows that current methods result in biased estimates of target ability when a speed-ability trade-off is present. The paper concludes with a discussion of ways in which the issue can be resolved.

show abstract

“…Therefore, further investigations were carried out as aimed at skipping response patterns because differences between high and low performers are exaggerated if low performers did not respond to any CR items due to operational scoring rules (e.g., Rose et al 2017). In present analysis, omitted responses before a valid response were treated as incorrect 5 following scoring rules that have been applied operationally in PISA since its inception.…”

Section: Resultsmentioning

confidence: 99%

Investigating Rater Effects in International Large-Scale Assessments

Shin

Davier

Yamamoto

2019

Theoretical and Practical Advances in Computer-Based Educational Measurement

Self Cite

View full text Add to dashboard Cite

The present study investigates rater effects in international large-scale assessments to evaluate the construct validity of constructed-response items scored by human raters. Using the Programme for International Student Assessment data collected in 2015, we illustrate the methods and present the findings about rater effects on constructed-response items in the context of the first fully computer-based international student skill survey. By comparing the unidimensional and the twodimensional multiple-group item response theory model, it was shown that the latent correlations between human-and machine-coded items were almost perfect. Country means of human-and machine-coded items were generally similar and their differences were small. Further investigation into individual rater effects mostly resulted in small random loadings, which implied that rater effects were negligible in most countries.

show abstract

Modeling Omitted and Not-Reached Items in IRT Models

Cited by 62 publications

References 40 publications

Predictors of individual performance changes related to item positions in PISA assessments

Predictors of individual performance changes related to item positions in PISA assessments

On the Importance of the Speed-Ability Trade-Off When Dealing With Not Reached Items

Investigating Rater Effects in International Large-Scale Assessments

Contact Info

Product

Resources

About