The Impact of Controlling for Extreme Responding on Measurement Equivalence in Cross-Cultural Research

Abstract

Prior research has shown that extreme response style can seriously bias responses to survey questions and that this response style may differ across culturally diverse groups. Consequently, cross-cultural differences in extreme responding may yield incomparable responses when not controlled for. To examine how extreme responding affects the cross-cultural comparability of survey responses, we propose and apply a multiple-group latent class approach where groups are compared on basis of the factor loadings, intercepts, and factor means in a Latent Class Factor Model. In this approach a latent factor measuring the response style is explicitly included as an explanation for group differences found in the data. Findings from two empirical applications that examine the cross-cultural comparability of measurements show that group differences in responding import inequivalence in measurements among groups. Controlling for the response style yields more equivalent measurements. This finding emphasizes the importance of correcting for response style in cross-cultural research.

Mixture Random-Effect IRT Models for Controlling Extreme Response Style on Rating Scales

“…The threshold parameters were set to −0.6, 0, and 0.6 for the 4-category items and −0.8, −0.4, 0, 0.4, and 0.8 for the 6-category items. The specifications of the model parameters were consistent with those that are commonly found in practice and similar to previous research (e.g., Li et al, 2006; Morren et al, 2012; Jin and Wang, 2014a; Huang, 2014, 2015, 2016). Each condition was replicated 30 times.…”

“…In turn, these problems can confound the nuisance ERS effect and the measure-intended dimension (Cheung and Rensvold, 2000; Bolt and Johnson, 2009; Thissen-Roe and Thissen, 2013). In fact, if controls for ERS are absent, inferences regarding cross-cultural comparisons and measurement equivalence are misleading (Morren et al, 2012). Under real testing situations, differences in ERS tendencies among respondents and differences in item parameters among response styles may occur simultaneously; thus, their effects should be distinguished in a latent trait model.…”

“…Because different response styles may have different response patterns, mixture IRT models can be used to classify respondents with respect to their response styles. Mixture IRT models have been applied to analyze ERS data (e.g., Moors, 2008; Morren et al, 2012); however, existing models show some limitations in measuring the ERS degree for respondents and lack a justification for the assumption of zero variability within latent classes with respect to the ERS levels (see below for a further description). Thus, a new mixture IRT model for ERS is required to simultaneously enable the classification and quantification of respondents with respect to the ERS tendency, evaluate latent DIF items that are caused by the ERS effect, and model the effects of item characteristics that may elicit ERS on item responses as a function of the measure-intended latent trait.…”

“…Morren et al (2012) extended the MNRM for ERS (e.g., Bolt and Newton, 2011) using latent classes to identify three types of response styles (normal, ERS, and MRS) and allowed different latent means with respect to the ERS tendency in the three latent classes. This model attempted to combine a nominal response model with a finite latent class model and quantify the ERS tendency.…”

Validity of Multiprocess IRT Models for Separating Content and Response Styles

“…Some authors attempted to control for response styles by either studying ipsative measures or standardized individual scores (e.g., M. W.-L. Cheung & Chan, 2002;Fischer, 2004) or by conducting analyses of partial correlations (e.g., ten Berge, 1999). Moreover, researchers made use of structural equation models (e.g., Billiet & McClendon, 2000), item response models (e.g., De Jong et al, 2008;Jin & Wang, 2013;Johnson & Bolt, 2010), latent class or mixed Rasch models (e.g., Gollwitzer, Eid, & Jürgensen, 2005;Meiser & Machunsky, 2008;Morren, Gelissen, & Vermunt, 2012), and other, more specialized approaches (e.g., Camparo & Camparo, 2013;Johnson, 2003;Rossi, Gilula, & Allenby, 2001). Finally, Weijters and colleagues employed sophisticated factorial models in the framework of structural equation modeling (e.g., Weijters, Geuens, et al, 2010a).…”