Reducing Cross-Cultural Comparability Bias Obtained with Measurement Invariance Analysis by Means of Anchoring Vignettes in Heterogeneous Refugee Samples

Abstract

Background: Configural, metric, and scalar measurement invariance have been indicators of bias-free statistical cross-group comparisons, although they are difficult to verify in the data. Low comparability of translated questionnaires or the different use of response formats by respondents might lead to rejection of measurement invariance and point to comparability bias in studies that use different languages. Anchoring vignettes have been proposed as a method to control for the different use of response formats by respondents (RC-DIF) as implemented by means of rating scales. We evaluate the question whether the comparability bias obtained by means of measurement invariance analysis can be reduced by means of anchoring vignettes or by considering socio-demographic heterogeneity as an alternative approach. Methods: We use the Health System Responsiveness (HSR) questionnaire in English and Arabic in a refugee population. We collected survey data in English (n = 183) and Arabic (n=121) in a random sample of refugees in the third largest German federal state. We conducted multiple-sample Confirmatory Factor Analyses (MGCFA) to analyse measurement invariance and compared the results when 1) using rescaled data on the basis of anchoring vignettes (non-parametric approach), 2) including information on DIF from the analyses with anchoring vignettes as covariates (parametric approach) and 3) including socio-demographic covariates. Results: For the HSR, every level of measurement invariance between Arabic and English questionnaires was rejected. Implementing rescaling on the basis of anchoring vignettes provided superior results over the initial MGCFA analysis, since configural, metric and scalar invariance could not be rejected. When using solely socio-demographic covariates, scalar measurement invariance could not be rejected, but configural and metric invariance had to be rejected. Conclusions: Surveys may consider anchoring vignettes as a method to obtain more satisfactory results of measurement invariance analyses; however, socio-demographic information cannot be included in the models as a standalone method. More research on the efficient implementation of anchoring vignettes and further development of methods to incorporate them when modelling measurement invariance is needed.