Making sense out of measurement non-invariance: how to explore differences among educational systems in international large-scale assessments

Cuellar, Edwin; Partchev, Ivailo; Zwitser, Robert; Bechger, Timo M.

doi:10.1007/s11092-021-09355-x

Cited by 5 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In such a situation, it may be more rewarding to study measurement invariance than merely try to overcome it. On the methodological size, conceptualizing DIF has proved to be more challenging than expected, such that there is still active work on the topic [32].…”

Section: Measurement Invariancementioning

confidence: 99%

dexter: An R Package to Manage and Analyze Test Data

Partchev

Koops

Bechger³

et al. 2023

Psych

Self Cite

View full text Add to dashboard Cite

In this study, we present a package for R that is intended as a professional tool for the management and analysis of data from educational tests and useful both in high-stakes assessment programs and survey research. Focused on psychometric models based on the sum score as the scoring rule and having sufficient statistics for their parameters, dexter fully exploits the many theoretical and practical advantages of this choice: lack of unnecessary assumptions, stable and fast estimation, and powerful and sensible diagnostic techniques. It includes an easy to use data management system tailored to the structure of test data and compatible with the current paradigm of tidy data. Companion packages currently include a graphical user interface and support for multi-stage testing.

show abstract

Section: Measurement Invariancementioning

confidence: 99%

dexter: An R Package to Manage and Analyze Test Data

Partchev

Koops

Bechger³

et al. 2023

Psych

Self Cite

View full text Add to dashboard Cite

show abstract

“…For instance, various researchers have proposed variations of (mixed) (generalized) linear regression models in which item- and person-specific covariates can be tested to explain (part of) the DIF effect (e.g., Balluerka et al, 2014; Van den Noortgate & De Boeck, 2005; Zumbo & Gelin, 2005). Other statistical approaches are based on analysis of the item distractor options (Penfield, 2010; Tsaousis et al, 2018), singular value decomposition (Cuellar et al, 2021), and mixture analysis (De Boeck et al, 2011). In addition, qualitative approaches include interviews (Benítez & Padilla, 2014) and experimental research (Wicherts et al, 2005).…”

mentioning

confidence: 99%

Relating violations of measurement invariance to group differences in response times.

Molenaar,

Feskens

2024

Psychological Methods

View full text Add to dashboard Cite

Validity of test score interpretations and cross-cultural comparisons in the First and Second International Science Studies

Sosa Paredes,

Andersson

2024

Educ Asse Eval Acc

View full text Add to dashboard Cite

In international large-scale assessments, student performance comparisons across educational systems are frequently done to assess the state and development in different domains. These results often have a large impact on educational policy and on the perceptions of an educational system’s performance. Early assessments, such as the First and Second International Science Studies (FISS and SISS), have been used alongside recent studies to create unique scales for investigating changes in constructs. The implicit assumptions in system comparisons are that the measures are valid, reliable, and comparable. However, these assumptions have not always been investigated thoroughly. This study aims to investigate the validity and cross-system comparability of scores from the FISS and SISS, conducted by the International Association for the Evaluation of Educational Achievement in 1970–1971 and 1983–1984. Findings based on item response theory (IRT) modeling indicate that scores in most educational systems can be viewed as reliable measures of a single science construct, supporting the validity of test score interpretations in these educational systems individually. In a robust assessment of measurement invariance using standard IRT methods, an alignment-based method, and the root mean square difference (RMSD) fit statistic, we demonstrate that measurement invariance is violated across systems. The alignment-based method identified a well-fitting model with complex restrictions but no items exhibited invariance across all systems, a result supported by the RMSD statistics. These results question the appropriateness of score comparisons across systems in FISS and SISS. We discuss the implications of these results and outline consequences for score comparisons across time.

show abstract

Making sense out of measurement non-invariance: how to explore differences among educational systems in international large-scale assessments

Cited by 5 publications

References 21 publications

dexter: An R Package to Manage and Analyze Test Data

dexter: An R Package to Manage and Analyze Test Data

Relating violations of measurement invariance to group differences in response times.

Validity of test score interpretations and cross-cultural comparisons in the First and Second International Science Studies

Contact Info

Product

Resources

About