Item order and speededness: implications for test fairness in higher educational high-stakes testing

Becker, Benjamin J.; Rijn, Peter van; Molenaar, Dylan; Debeer, Dries

doi:10.1080/02602938.2021.1991273

Cited by 3 publications

(3 citation statements)

References 38 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our findings might only be generalizable to other low-stakes assessment studies like PISA [ 28 , 29 , 30 ]. However, the underlying mechanisms for missing item responses can strongly differ from high-stakes assessment studies [ 31 ].…”

Section: Introductionmentioning

confidence: 99%

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Robitzsch

2021

EJIHPE

View full text Add to dashboard Cite

Missing item responses are prevalent in educational large-scale assessment studies such as the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians have advocated for a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. In an illustrative simulation study, it is shown that the Mislevy-Wu model provides unbiased model parameters. Moreover, the simulation replicates the finding from various simulation studies from the literature that scoring missing item responses as wrong provides biased estimates if the latent ignorability assumption holds in the data-generating model. However, if missing item responses are generated such that they can only be generated from incorrect item responses, applying an item response model that relies on latent ignorability results in biased estimates. The Mislevy-Wu model guarantees unbiased parameter estimates if the more general Mislevy-Wu model holds in the data-generating model. In addition, this article uses the PISA 2018 mathematics dataset as a case study to investigate the consequences of different missing data treatments on country means and country standard deviations. Obtained country means and country standard deviations can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, in the discussion section, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

show abstract

Section: Introductionmentioning

confidence: 99%

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Robitzsch

2021

EJIHPE

View full text Add to dashboard Cite

show abstract

“…In high-stakes assessments, for example, examinees may copy answers from other test takers, exhibit poor time management, or apply item preknowledge (Becker, Van Rijn, Molenaar, & Debeer, 2021;Belov, 2011Belov, , 2016Hong & Cheng, 2019;Sinharay, 2017a;X. Wang, Liu, & Hambleton, 2017).…”

Section: Introductionmentioning

confidence: 99%

Online Proctoring Based on Item Responses and Response Times

Han¹,

Kang²

2022

Preprint

View full text Add to dashboard Cite

The surge of online testing in recent years calls for procedures that monitor examinees’ test-taking behaviors in real time. This study presents online proctoring methods that examine response and response-time behaviors during testing. The procedures are developed using sequential quality control techniques and implemented with various sampling designs to improve time sensitivity. Experimentation with simulated data suggests that the proposed methods can provide a powerful online surveillance tool for regulating anomalous behaviors. The procedures showed high sensitivity to aberrances and gave timely signals while maintaining the error rates reasonably small. We demonstrate utility of the procedures using real-assessment data and discuss implications for practical use.

show abstract

“…The first step in gifted education is identifying students through the use of talent and creativity tests. As such, the psychometric properties of these tests used in gifted education screening and identification have important implications for multiple stakeholders especially in high-stakes testing (Becker, Van Rijn, Molenaar, & Debeer, 2021). Specifically, the main psychometric property of interest is how the test as a whole, and components measuring cognitive subskills, are able to discriminate ability levels with respect to the skills being measured.…”

Section: Introductionmentioning

confidence: 99%

Differences in Discrimination with Respect to Latent Trait and Test-Taking Speed Across Items Measuring Sub-Domains of Cognitive Ability

Vista

Alahmadi

2022

Journal of Psychoeducational Assessment

View full text Add to dashboard Cite

The relationship between latent trait and test-taking speed is an important area of study in assessment research. In addition to contributions of such studies to psychometrics, the factors that affect both ability and speed have implications for test development and have policy consequences especially if the tests are high stakes. This study investigated the main assessment scale used in gifted education screening in the Kingdom of Saudi Arabia and focused on the discriminating properties of items across the several cognitive subskills being measured. Under an item response theoretic framework, this study used a lognormal response time model to investigate the discrimination parameter estimates for both ability and speed, with guessing taken into account. Results showed that cognitive subskills do not have uniform discrimination properties across ability and speed, such that some types of items are better at discriminating ability while other types are better at discriminating speed. At a test-level, our results also showed that ability is negatively correlated with speed, such that higher ability students tend to take more time and hence have slower speed. This finding supports other results in the literature. Implications for high-stakes testing and test development are discussed.

show abstract

Item order and speededness: implications for test fairness in higher educational high-stakes testing

Cited by 3 publications

References 38 publications

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Online Proctoring Based on Item Responses and Response Times

Differences in Discrimination with Respect to Latent Trait and Test-Taking Speed Across Items Measuring Sub-Domains of Cognitive Ability

Contact Info

Product

Resources

About