A comparison of question order effects on item-by-item and grid formats: visual layout matters

Abstract

Question order effect refers to the phenomenon that previous questions may affect the cognitive response process and respondents’ answers. Previous questions generate a context or frame in which questions are interpreted. At the same time, in online surveys, the visual design may also shift responses. Past empirical research has yielded considerable evidence supporting the impact of question order on measurement, but few studies have investigated how question order effects vary with the visual design. Our main research question was whether question order effects are different on item-by-item formats compared to grid formats. The study uses data from an online survey experiment conducted on a non-probability-based online panel in Hungary, in 2019. We used the welfare-related questions of the 8’th wave of ESS. We manipulated the questionnaire by changing the position of a question that calls forth negative stereotypes about such social benefits and services. We further varied the visual design by presenting the questions in separate pages (item-by-item) or one grid. The results show that placing the priming questions right before the target item significantly changed respondents’ attitudes in a negative way, but the effect was significant only when questions were presented on separate pages. A possible reason behind this finding may be that respondents engage in a deeper cognition when questions are presented separately. On the other hand, the grid format was robust against question order, in addition, we found little evidence of stronger satisficing on grids. The findings highlight that mixing item-by-item and grids formats in online surveys may introduce measurement inequivalence, especially when question order effects are expected.

Willingness to take responsibility: Self-sacrifice versus sacrificing others in takeover decisions during autonomous driving

“…Third, our participants answered the questions in a fixed sequence, which may have induced an order effect. However, previous studies have shown order effects in online surveys when questions were asked on separate pages [ 59 ]. In our survey, questions were presented on the same page one after the other, and we therefore assumed that the potential influence of order was small.…”

Question order effects: how robust are survey measures on political solidarities with reference to Germany and Europe?

“…Especially since citizen relationships towards these two entities have been politicized by political actors since the 1990s (Hooghe & Marks, 2005 ), even though Germany falls into the geographic and political area of Europe. This can result in question order effects due to a so-called part–part comparison in which the first question sets a frame of reference or standard of comparison for the following question (Silber et al, 2016 ; Stefkovics & Kmetty, 2022 ; Tourangeau et al, 2000 ). In the case of a German-European question order, it is assumable that respondents compare Europe to Germany (their home country) when evaluating Europe.…”

“…By unraveling the cognitive process of responding to survey questions in its main components (i.e. comprehension, retrieval, judgment, and response), question order effects can be associated with the information retrieval and judgement formation (Tourangeau et al, 2000 ) and occur when information and judgments of previous questions resonate with later questions (Stefkovics & Kmetty, 2022 ). This results in measurement error reducing data quality and weakens the significance of the conclusions that can be drawn from survey data.…”

“…Considering the survey literature, there are various empirical studies and scientific contributions on question order effects (see, for example, Schuman & Presser, 1981 ; Schwarz, 1991 ; Silber et al, 2016 ; Stark et al, 2020 ; Stefkovics & Kmetty, 2022 ; Sudman et al, 1996 ; Tourangeau et al, 2000 ). However, these studies mainly consider people’s response behavior in terms of response distributions.…”