Invariance analyses in large-scale studies

Vijver, Fons van de; Avvisati, Francesco; Davidov, Eldad; Eid, Michael; Fox, J-P.; Donné, Noémie Le; Lek, Kimberley; Meuleman, Bart; Paccagnella, Marco; Schoot, Rens van de

doi:10.1787/254738dd-en

Cited by 17 publications

(13 citation statements)

References 58 publications

(74 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rens van de Schoot suggested that because of the dependence on priors, practitioners should conduct a sensitivity analysis before drawing substantive conclusions, i.e. estimate models with different priors and verify the robustness of the resulting claims (Lek & van de Schoot, 2019). In general, there was no consensus on how to rank models based on different priors (and thus, select the "best" priors and models): Jean-Paul Fox highlighted that criteria such as posterior predictive p values (PPP) or deviance information criteria (DIC) should not be used to compare models with the same number of parameters.…”

Section: Bayesian Approximate Invariance Methodsmentioning

confidence: 99%

“…In response to some of these shortcomings, Jean-Paul Fox presented an alternative approach to assess whether the data support full invariance or only approximate invariance of measurements, which he illustrated in the IRT case (Fox, 2019). The approach, which was recently presented in Fox, Mulder, and Sinharay (2017), is based on the intuition that the marginal model obtained by integrating out the random parameters from a one-parameter IRT model is simply a fixed-effect model with a particular structure for the covariance of residuals.…”

Section: Bayesian Approximate Invariance Methodsmentioning

confidence: 99%

“…A second strategy, which can be described as "topdown", is better suited for international surveys with dozens of countries (Eid, 2019). In this strategy, the exploratory step to determine the optimal number of classes is conducted directly on the pooled dataset, assuming, in a first step, that only class membership (and not country membership) determines the response patterns, while group membership only influences the size of classes.…”

Section: Dealing With Imperfect Comparability Of Measurements When Scmentioning

confidence: 99%

See 2 more Smart Citations

A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys

Avvisati

Donné

Paccagnella

2019

Meas Instrum Soc Sci

View full text Add to dashboard Cite

The value of crosscountry comparisons is at the heart of large-scale international surveys. Yet the validity of such comparisons is often challenged, particularly in the case of latent traits whose estimates are based on self-reported answers to a small number of questionnaire items. Many believe self-reports to be unreliable and not comparable, and indeed, formal statistical procedures very often reject the assumption that the questions are understood and answered in the same way in different countries (measurement invariance). A methodological conference on the comparability of questionnaire scales was hosted by the OECD on 8 and 9 November 2018. This meeting report summarises the discussions held at the conference about measurement invariance testing and instrument design. The report first provides a brief introduction to the measurement models and the accompanying invariance analyses typically used in the industry of large-scale international surveys and points to the main limitations of these current standard approaches. It then presents classical and novel ways to deal with imperfect comparability of measurements when scaling and reporting on continuous traits and on categorical latent variables. It finally discusses the extent to which item design can improve the crosscountry comparability of the measured constructs (e.g. by adopting innovative item formats such as anchoring vignettes and situational judgement test items). It concludes with some general considerations for survey design and reporting on invariance analyses and survey results.

show abstract

Section: Bayesian Approximate Invariance Methodsmentioning

confidence: 99%

Section: Bayesian Approximate Invariance Methodsmentioning

confidence: 99%

Section: Dealing With Imperfect Comparability Of Measurements When Scmentioning

confidence: 99%

See 1 more Smart Citation

A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys

Avvisati

Donné

Paccagnella

2019

Meas Instrum Soc Sci

View full text Add to dashboard Cite

show abstract

“…The general idea is that an item-group effect can be detected as an additional correlation, when the assumption of measurement invariance is violated. This innovative Bayesian method for measurement invariance testing has also been discussed in Van de Vijver et al (2019) for binary data in which a comparison is made with the Mantel-Haenszel test.…”

Section: Introductionmentioning

confidence: 99%

Bayesian covariance structure modelling for measurement invariance testing

et al. 2020

View full text Add to dashboard Cite

In a Bayesian Covariance Structure Model (BCSM) the dependence structure implied by random item parameters is modelled directly through the covariance structure. The corresponding measurement invariance assumption for an item is represented by an additional correlation in the item responses in a group. The BCSM for measurement invariance testing is defined for mixed response types, where the additional correlation is tested with the Bayes factor. It is shown that measurement invariance can be tested simultaneously across items and thresholds for multiple groups. This avoids the risk of capitalization on chance that occurs in multiple-step procedures and avoids cumbersome procedures where items are examined sequentially. The proposed measurement invariance procedure is applied to PISA data, where the advantages of the method are illustrated.

show abstract

“…In practice, it is often assumed that item parameters are equal across groups, which is denoted as invariance. The invariance concept has been very prominent in psychology and the social sciences in general [1,2]. For example, in international large-scale assessment studies in education like the programme for international student assessment (PISA), the necessity of invariance is strongly emphasized [3].…”

Section: Introductionmentioning

confidence: 99%

$L_p$ Loss Functions in Invariance Alignment and Haberman Linking

Robitzsch¹

2020

Preprint

View full text Add to dashboard Cite

The comparison of group means in latent variable models plays a vital role in empirical research in the social sciences. The present article discusses extensions of invariance alignment and Haberman linking concerning the choice of linking functions for comparisons of many groups. Robust linking functions are proposed for invariance alignment and robust Haberman linking that are particularly suited to item response data under partial invariance. In a simulation study, it is shown that both linking approaches have comparable performance, and in some conditions, the newly proposed robust Haberman linking outperforms invariance alignment.

show abstract

Invariance analyses in large-scale studies

Cited by 17 publications

References 58 publications

A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys

A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys

Bayesian covariance structure modelling for measurement invariance testing

$L_p$ Loss Functions in Invariance Alignment and Haberman Linking

Contact Info

Product

Resources

About