How IRT can solve problems of ipsative data in forced-choice questionnaires.

Abstract

In multidimensional forced-choice (MFC) questionnaires, items measuring different attributes are presented in blocks, and participants have to rank-order the items within each block (fully or partially). Such comparative formats can reduce the impact of numerous response biases often affecting single-stimulus items (aka, rating or Likert scales). However, if scored with traditional methodology, MFC instruments produce ipsative data, whereby all individuals have a common total test score. Ipsative scoring distorts individual profiles (it is impossible to achieve all high or all low scale scores), construct validity (covariances between scales must sum to zero), criterion related validity (validity coefficients must sum to zero), and reliability estimates.We argue that these problems are caused by inadequate scoring of forced-choice items, and advocate the use of item response theory (IRT) models based on an appropriate response process for comparative data, such as Thurstone's Law of Comparative Judgment. We show that by applying Thurstonian IRT modeling (Brown & Maydeu-Olivares, 2011), even existing forcedchoice questionnaires with challenging features can be scored adequately and that the IRTestimated scores are free from the problems of ipsative data. Assessments of personality, social attitudes, interests, motivation, psychopathology and well-being largely rely on respondent-reported measures. Most such measures employ the socalled single-stimulus format, where respondents evaluate one question (or item) at a time, often in relation to a rating scale (i.e. Likert-type items). Because the respondents rate each item separately from other items, they make absolute judgments about the extent to which the item describes their personality, attitudes, etc. Simple to answer and score and therefore popular with test takers and test users, the single-stimulus format makes several assumptions about the respondents' rating behaviors that are often unrealistic. For instance, the use of rating scales relies on the assumption that respondents interpret category labels in the same way. This assumption is very rarely tested in practice, but research available on the issue suggests that interpretation and meaning of response categories vary from one respondent to another (Friedman & Amoo, 1999). Furthermore, individual response styles may vary (Van Herk, Poortinga & Verhallen, 2004) so that some respondents avoid extreme categories (central tendency responding), whereas others prefer them (extreme responding). Sometimes respondents tend to agree with both positive and negative statements as presented (acquiescence bias).Another common problem is getting respondents to differentiate between ratings they give to single-stimulus items. When rating another person's attributes or behavior (as in the 360-degree feedback), respondents commonly give either high or low ratings on all behaviors (halo/horn effect) depending on whether they judge the person to score high or low on a single important dimension. Typically, respon...

Equivalence of Narcissistic Personality Inventory constructs and correlates across scoring approaches and response formats

“…However, a recent study suggests that some item pairs in the NPI may be multidimensional (Ackerman et al, 2015), in which case partially ipsative scores are obtained (A. Brown & Maydeu-Olivares, 2013 One obvious solution to the partially ipsative data of the NPI is to model the responses using appropriate forced-choice models, which can incorporate both the unidimensional and multidimensional item pairs.…”

Forced-Choice Assessment of Work-Related Maladaptive Personality Traits: Preliminary Evidence From an Application of Thurstonian Item Response Modeling

“…Another challenge is that classical scoring of forced-choice questionnaires results in the same total score for every individual -so-called ipsative data (Clemans, 1970;Meade, 2004). Problems of ipsative data include biased reliability estimates, biased estimates of construct and criterionrelated validity due to the constraint that covariances of all traits necessarily sum to zero, and biased individual scores (Brown & Maydeu-Olivares, 2013).…”

Personality Assessment, Forced-Choice

“…This improvement is presumably due to reduction in response biases detrimental to validity. The psychometric problems of ipsative scores, however, should not be underestimated as they can result in spuriously significant validity coefficients (Brown and Maydeu-Olivares, 2013). It is my belief that conclusive validity evidence for forced-choice assessments can only be gained by using model-based measurement.…”

“…Principal Components analysis can be applied to ipsative scores; however, the results are difficult to interpret since the components tend to contain conflicting rather than convergent traits (Cornwell and Dunlap, 1994); 3) Ipsative scores distort criterion-related validity estimates since their correlations with any external measure must sum to 0, creating spuriously positive and negative correlations; 4) Internal consistency reliability cannot be applied since the assumption of consistent coding is not met in ipsative scores (Brown and Maydeu-Olivares, 2013). …”

“…Applications of this flexible model have included re-analysis of existing forced-choice questionnaire data and development of new measures. Re-analysis of the Customer Contact Styles Questionnaire data (CCSQ) demonstrates the advantages of IRT modeling for personality assessment; specifically, interpersonal comparability of person trait scores estimated by the IRT method as opposed to the classical method resulting in ipsative scores (Brown and Maydeu-Olivares, 2013). The development of a new IRT scored version of the Occupational Personality Questionnaire (OPQ32r) illustrates how Thurstonian IRT modeling may be applied to re-analyze and re-develop an existing assessment tool, enhancing its strong features and transforming its scoring protocol (Brown and Bartram, 2009).…”