First Equals Most Important? Order Effects in Vignette-Based Measurement

Abstract

Standard-Nutzungsbedingungen:Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden.Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen.Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in First equals most important? Order effects in vignette-based measurement Non-technical summaryVignettes are increasingly used in social science surveys, to measure how people make decisions and what determines their attitudes. A vignette typically describes a hypothetical situation or object, about which respondents are asked to make a judgment. The object is described as having various characteristics. For example in our research, the vignettes describe full-time employees. The characteristics of the vignettes are experimentally varied, so that researchers can estimate the impact of individual characteristics on respondents'judgments. In our research we use vignettes to investigate which characteristics of an employee, or the organization they are working for, should determine how much they earn, in order for their earnings to be judged as fair. Our results show that strong order effects can occur, which alter conclusions about which characteristics respondents think should determine how much an employee earns. The order however only matters when the vignettes are complex, that is, when employees are described with 12 rather than 8 different characteristics -or when respondents are asked two questions about each vignette rather than just one. Order effects are more likely for respondents who have little knowledge or weak attitudes about the topic the vignettes are describing. Contrary to expectations respondents' cognitive ability did not appear to matter. The results have implications for how best to design vignettes, in order to avoid order effects that could impact results. First equals most important?Order effects in vignette-based measurement Katrin Auspurg (University of Konstanz) and Annette Jäckle (University of Essex) AbstractA vignette typically describes a hypothetical situation or object which respondents are asked to judge. The object is described as having different dimensions, the values of which are experimentally varied, so that their impact on respondents' judgments can be estimated. We examine 1) whether the order in which dimensions are presented impacts estimates, and 2) under which conditions order effects are mostly likely. Using data from a web survey of students we analyze several possible conditions: features of the vignette design, characteristics of respondents, and interactions between these. Our results show that strong order effects can occur, but only when the vignettes are complex.

The Effects of Vignette Placement on Attitudes Toward Supporting Family Members

“…Some studies find that context effects are greater among people with weak attitudes (Schwarz et al 1991). Auspurg and Jäckle (2012) find that individuals with weak attitudes are more susceptible to order effects of characteristics within a vignette. Other studies find no association between context and attitude strength (see review by Tourangeau et al 2003).…”

Attribute level overlap (and color coding) can reduce task complexity, improve choice consistency, and decrease the dropout rate in discrete choice experiments

“…Interestingly, we do find that attribute level overlap and color coding had an impact on the estimated preference structure: Attribute level overlap reduced the emphasis that respondents place on the first and last attribute in the DCE. Without overlap, the first and last attribute typically receive the most attention from respondents, see, for example, Chrzan (), Kjær, Bech, Gyrd‐Hansen, and Hart‐Hansen () and Auspurg and Jäckle (), and the introduction of overlap subsequently forces respondents to take all attributes into account and thus more equally divides their attention over all included attributes. Intensity color coding increased the distance between “slight” and “moderate” problems, which are notoriously difficult to distinguish in the Dutch translation of the EQ‐5D and more clearly separated when color coding was used. Given that illogical preference reversals were observed in the DCE results for three of the five dimensions in the official Dutch EQ‐5D valuation study for precisely these levels (Versteegh et al, ) and because color coding had no impact on the overall range of the preference estimates, we are inclined to consider this a beneficial outcome of color coding. Highlighting of differences reduced the emphasis placed on the moderate levels and increased the emphasis placed on the more extreme levels.…”

From the general to the specific: How social trust motivates relational trust

“…3 Vignette methodologists find that respondents perform well with ten vignettes per person and no more than twelve randomized dimensions per vignette (Auspurg and Jäckle 2015;Sauer et al 2011). With this design, the number of vignettes and dimensions is small enough to avoid biases that might jeopardize validity and reliability, such as cognitive overload, learning effects, and order effects, but the number of vignettes is large enough to explore within-and between-individual variation in the outcome variables of interest.…”