Validity of Multiprocess IRT Models for Separating Content and Response Styles

Abstract

Response styles, the tendency to respond to Likert-type items irrespective of content, are a widely known threat to the reliability and validity of self-report measures. However, it is still debated how to measure and control for response styles such as extreme responding. Recently, multiprocess item response theory models have been proposed that allow for separating multiple response processes in rating data. The rationale behind these models is to define process variables that capture psychologically meaningful aspects of the response process like, for example, content-and response style-related processes. The aim of the present research was to test the validity of this approach using two large data sets. In the first study, responses to a 7-point rating scale were disentangled, and it was shown that response style-related and content-related processes were selectively linked to extraneous criteria of response styles and content. The second study, using a 4-point rating scale, focused on a content-related criterion and revealed a substantial suppression effect of response style. The findings have implications for both basic and applied fields, namely, for modeling response styles and for the interpretation of rating data.

A New Model for Acquiescence at the Interface of Psychometrics and Cognitive Psychology

“…In summary, the Böckenholt Model assumes three qualitatively distinct processes to account for MRS, ERS, and the target trait. Evidence for the validity of this approach comes from the theoretical derivation of the model based on underlying cognitive processes (Böckenholt, 2012a) as well as from empirical data: For example, the construct validity of the three processes was demonstrated by Hansjörg Plieninger and Meiser (2014) in a study using extraneous style-and contentrelated criteria. Khorramdel and von Davier (2014) extended the model to questionnaires with multiple domains (Big Five) and were able to show that MRS and ERS are stable across different scales.…”

“…The Böckenholt Model of response styles and IR-tree models in general are characterized by these definitions of conditional response processes similar as in MPT models that are used in cognitive psychology (e.g., Erdfelder et al, 2009;Matzke et al, 2015). In psychometrics, this approach has been proven useful in both methodological and applied work (e.g., Böckenholt, 2017;Böckenholt & Meiser, 2017;Jeon & De Boeck, 2016;Khorramdel & von Davier, 2014;Hansjörg Plieninger & Meiser, 2014;Thissen-Roe & Thissen, 2013;Zettler, Lang, Hülsheger, & Hilbig, 2016). However, ARS-which is often seen as an especially important response style (e.g., Hansjörg Plieninger, 2017; Rammstedt, Goldberg, & Borg, 2010)-was not included in the Böckenholt Model, a disadvantage compared to alternative response style models (e.g., Johnson & Bolt, 2010).…”

A generalized item response tree model for psychological assessments

“…Differentiating slow and fast intelligence , modeling motivated misreports to sensitive survey questions (Boeckenholt, 2013), examining content and response styles in multiple-choice items (Plieninger & Meiser, 2014), and modeling skipped and non-reached item responses (Debeer et al, submitted) are some of the examples of how item response tree models have been utilized. These prior approaches, however, have some limitations.…”

Mixture Random-Effect IRT Models for Controlling Extreme Response Style on Rating Scales

“…Under these circumstances, we say that the individual exhibits a particular response style. Such styles can potentially distort the reliability and validity of investigative measures (De Jong et al, 2008; De Beuckelaer et al, 2010; Bolt and Newton, 2011; Plieninger and Meiser, 2014). Various response styles have been noted (Baumgartner and Steenkamp, 2001), which are assumed to be independent of the item content (Nunnally, 1978; Paulhus, 1991) and to be stable respondent characteristics over time (Berg, 1953; Hamilton, 1968; Bachman and O'Malley, 1984; Weijters et al, 2010).…”

“…Among these response styles, extreme response style (ERS), which is defined as a systematic tendency to select the end points of a rating scale, has attracted significant research attention (Hamilton, 1968; Greenleaf, 1992; Bolt and Johnson, 2009; Johnson and Bolt, 2010; Weijters et al, 2010; Bolt and Newton, 2011; Thissen-Roe and Thissen, 2013; Van Vaerenbergh and Thomas, 2013; Khorramdel and von Davier, 2014; Plieninger and Meiser, 2014; Jin and Wang, 2014a). Therefore, ERS and its opposite effect, mild response style (MRS), which is defined as a high frequency of endorsing middle responses of rating-scale items, are the focus of this study.…”