Generalization of Classic Question Order Effects Across Cultures

Abstract

Questionnaire design is routinely guided by classic experiments on question form, wording, and context conducted decades ago. This article explores whether two question order effects (one due to the norm of evenhandedness and the other due to subtraction or perceptual contrast) appear in surveys of probability samples in the United States and 11 other countries (Canada, Denmark, Germany, Iceland, Japan, the Netherlands, Norway, Portugal, Sweden, Taiwan, and the United Kingdom; N = 25,640). Advancing theory of question order effects, we propose necessary conditions for each effect to occur, and found that the effects occurred in the nations where these necessary conditions were met. Surprisingly, the abortion question order effect even appeared in some countries in which the necessary condition was not met, suggesting that the question order effect there (and perhaps elsewhere) was not due to subtraction or perceptual contrast. The question order effects were not moderated by education. The strength of the effect due to the norm of evenhandedness was correlated with various cultural characteristics of the nations. Strong support was observed for the form-resistant correlation hypothesis.

Question order bias revisited: A split‐ballot experiment on satisfaction with public services among experienced and professional users

“…Although such methodological concerns should not dictate which populations are sampled, they do suggest that one needs to reflect a priori on whether a particular sample consists of respondents with low expertise on the topics examined. Still, as indicated by our results and recent cross‐national evidence (Stark et al 2018), even well‐educated experts who are asked about something they encounter in their routine work can be subject to question order bias. Again, we therefore suggest that question order is randomized to provide an estimate of bias in the sample at hand.…”

“…Even in a high‐expertise sample, we find an effect that is similar in size to previous estimates. This suggests that question order bias in satisfaction measures is more universal than is sometimes assumed (see also Stark et al 2018). Our experimental results gained additional and unanticipated leverage.…”

“…The fifth, and final, factor often highlighted is the background of the respondents themselves (McFarland 1981, p. 209; Stark et al 2018). Their educational level, occupation, broad political knowledge and expertise on the topics in question are stressed as important to context effects (Van de Walle and Van Ryzin 2011, p. 1439).…”

“…When survey questions stand in a part‐whole relationship to each other and their sequence is either specific‐general or general‐specific, various studies have found both assimilation and contrast effects (e.g., Schuman and Presser 1981; Schwarz et al 1991; Kaplan et al 2013; Stark et al 2018). This inconsistency probably has to do with variations in the number of specific questions and conversational norms.…”

Question Order Effects in Cross-Cultural Web Probing: Pretesting Behavior and Attitude Questions

“…In the aftermath, the applicability of the underlying conversational norms (Grice, 1975) to collectivist cultures has been questioned (Schwarz et al, 2010). In a comparative study involving 11 countries, Stark et al (2018) find that both differences in survey response styles and topic-specific differences contribute to explaining differing question order effects across countries. Topic-specific differences between countries may impact question order effects for attitude and behavior questions if, for instance, a certain behavior is more common or considered to be more acceptable in one country.…”