Order Effects in Complex and Simple Tasks

Abstract

There is strong evidence that the order in which response options are presented in surveys significantly affects the answers that respondents provide. According to the theory of survey satisficing, the severity of order effects should increase with task difficulty. However, the tasks provided to respondents in existing studies of response-order effects are generally very simple, making it difficult to evaluate the satisficing hypothesis. Further, evidence from cognitive psychology suggests a completely different mechanism: people are more motivated to persist in completing tasks when they are intricate, challenging, and enriching. I designed survey experiments administered over the Internet consisting of two types of tasks: (1) a complex task in which respondents were asked to rank seven public officials in order of how much they should be blamed for the property damage and loss of life caused by Hurricane Katrina in the city of New Orleans; and (2) a series of simple tasks in which respondents answered items with ordinal response choices on rating scales. I found almost no order effects in the complex task among all educational groups. Conversely, I found significant and substantial order effects in the simple tasks, particularly among low-education respondents. These results suggest that theories of survey satisficing may simplify matters by assuming that satisficing monotonically increases with task difficulty. Moreover, my findings have important implications for questionnaire design, underscoring the importance of randomizing response options. A principal concern of survey methodologists is whether the order in which response options are presented affects the data that respondents produce (Schuman and Presser 1996). In a meta-analysis of several studies of order effects in categorical response choices, Krosnick (1999) and Krosnick and

Plate shape and colour interact to influence taste and quality judgments

“…Malhotra [35] showed that ratings on simple questions, questions that are not challenging and take little effort, were susceptible to order effects. Since all participants in this study completed the attribute ratings in the same order (that is, sweetness followed by intensity followed by quality followed by liking), it is possible that each subsequent rating was influenced by the previous leaving sweetness ratings the only purely independent rating of the four.…”

First Equals Most Important? Order Effects in Vignette-Based Measurement

“…Prior studies on the relationship between task difficulty and order effects suffer from the limitation that the response task differed not only in complexity but also in other features like the question format or answer scales (Malhotra 2009). Factorial surveys offer the possibility of varying complexity while keeping the question format and response options fixed.…”

Methodological Issues in Whistle-blowing Intentions Research: Addressing the Social Desirability Bias and Order Effect Bias

“…Methodologically, the choice of presentation order of the vignettes may significantly affect the respondents' answers (LaSalle, 1997;Malhotra, 2009). A type of bias, known as an order effect bias can thus affect the validity of the research instrument (Dillman, 2000).…”

“…The problem of order effects has been observed in a number of studies (see Hogarth and Einhorn, 1992;LaSalle, 1997;Malhotra, 2009), however, such problem received little attention in the whistle-blowing literature. None of whistle-blowing studies that utilised vignettes did address the issue of order effect bias despite utilising a set of vignettes sequentially in examining their respondents' whistle-blowing decisions.…”