Response Bias and Reliability in Verbal Agreement Rating Scales: Does Polarity and Verbalization of the Middle Category Matter?

Abstract

Verbal rating scale polarity and verbalizations of the middle category that do not match the polarity in agreement rating scales were investigated. Two randomized web survey experiments were conducted using a probability panel of German Internet users. The classical bipolar “disagree/agree” verbalization was compared with the unipolar “do not agree/agree” alternative. In both experiments, attitudes on gender roles were measured using a different number of rating scale categories (seven vs. five) in each. After controlling for variables related to satisficing, biases were lowest with the unipolar “do not agree/agree” rating scale in which the verbalization of the middle category was also unipolar. Reliability decreased when the verbalization of the middle category did not suit the rating scale polarity. Researchers should pay attention to the polarity of rating scales and use verbalization of the middle category consistent with the rating scale polarity.

Measuring Income (In)equality: Comparing Survey Questions With Unipolar and Bipolar Scales in a Probability-Based Online Panel

“…The decision of whether to measure respondents' attitudes and opinions with unipolar or bipolar scales is somewhat controversial (see, for instance, Alwin, 2007Alwin, , 2010DeCastellarnau, 2018;Krosnick & Fabrigar, 1997;Menold, 2019;Menold & Raykov, 2015;Schaeffer & Presser, 2003;Thomas & Barlas, 2018). Some researchers recommend the use of unipolar scales (e.g., running along an agreement continuum), and some others recommend the use of bipolar scales (e.g., running along an agreement and disagreement continuum).…”

“…The problem of comparability is especially associated with fully verbalized scales in which the bipolar labels of the midpoint and the negative (or opposite) part of the scale differ from the unipolar labels. While the unipolar midpoint (usually) indicates a moderate level of agreement, the bipolar midpoint can have different meanings depending on the respective wording (see Menold, 2019;Wang & Krosnick, 2019). For instance, it can be conceived as indicating ambivalence or indifference (Kennedy, 2008;Menold, 2019;O'Muircheartaigh et al, 1995;Schaeffer & Presser, 2003;Schneider & Schwarz, 2017).…”

“…For instance, it can be conceived as indicating ambivalence or indifference (Kennedy, 2008;Menold, 2019;O'Muircheartaigh et al, 1995;Schaeffer & Presser, 2003;Schneider & Schwarz, 2017). Ambivalence implies that respondents have positive and negative feelings preventing them from having a clear attitude or opinion toward the object under consideration (Menold, 2019;Schaeffer & Presser, 2003;Schneider & Schwarz, 2017). This particularly applies when using middle categories that consist of "partly/partly" formulations.…”

“…This particularly applies when using middle categories that consist of "partly/partly" formulations. In contrast, indifference implies that respondents either have a neutral or no position at all toward the object under consideration (Krosnick & Fabrigar, 1997;Menold, 2019;Sturgis et al, 2014). This particularly applies when using middle categories that consist of neither/nor formulations.…”

“…It seems that responses to bipolar scales, compared to responses to unipolar scales, are characterized by a shift toward the positive end of the scale (Kennedy, 2008;Krebs, 2012;Kunz, 2015;Menold, 2019). This shift manifests itself in a higher selection of positive (agree) response categories in bipolar scales than in unipolar ones (Kennedy, 2008;Menold, 2019;O'Muircheartaigh et al, 1995;Schaeffer & Presser, 2003). In contrast, it seems that responses to unipolar scales, compared to those to bipolar scales, are characterized by a shift toward the middle of the scale, which manifests itself in a higher selection of the middle category (O'Muircheartaigh et al, 1995).…”