Method Effects in Factorial Surveys: An Analysis of Respondents' Comments, Interviewers' Assessments, and Response Behavior

Abstract

This series presents research findings based either directly on data from the German SocioEconomic Panel Study (SOEP) or using SOEP data as part of an internationally comparable data set (e.g. CNEF, ECHP, LIS, LWS, CHER/PACO). SOEP is a truly multidisciplinary household panel study covering a wide range of social and behavioral sciences: economics, sociology, psychology, survey methodology, econometrics and applied statistics, educational science, political science, public health, behavioral genetics, demography, geography, and sport science.The decision to publish a submission in SOEPpapers is made by a board of editors chosen by the DIW Berlin to represent the wide range of disciplines covered by SOEP. There is no external referee process and papers are either accepted or rejected without revision. Papers appear in this series as works in progress and may also appear elsewhere. They often represent preliminary studies and are circulated to encourage discussion. Citation of such a paper should account for its provisional character. A revised version may be requested from the author directly.Any opinions expressed in this series are those of the author(s) and not those of DIW Berlin.Research disseminated by DIW Berlin may include views on public policy issues, but the institute itself takes no institutional policy positions. (1) respondents' comments about the vignettes, (2) interviewers' assessments of respondents comprehension and willingness to answer, and (3) response behavior regarding response time, use of the answering scale, and consistency of evaluations by different age groups and educational background. The respondents evaluated 24 vignettes consisting of ten dimensions that described full-time employees and their gross earnings. The evaluation task was to assess whether the given earnings were just or unjust, and if they were rated as unjust, respondents had to specify the amount of injustice on an 100-point rating scale. In regard to respondents' comments, the critique mentioned most frequently by respondents referred to the content (unrealistic descriptions) and the number of the vignettes. The analysis of the interviewers' assessments of respondents' comprehension and willingness to answer revealed less comprehension and willingness to answer for older and less well educated respondents, although these differences were similar to those found for the complete questionnaire. The analysis of the response behavior revealed no differences of response time between the groups. Analyses of response consistency show that one should consider hints for simplifying heuristics: such heuristics can lead to an artificially high response consistency. The implications of the findings are discussed.JEL: C81, D63, J31

Gender Bias in Justice Evaluations of Earnings: Evidence From Three Survey Experiments

“…The following sections briefly describe the respondent samples and provide an overview of the factorial surveys and additional variables used and the analysis technique employed. There are methods reports available that provide additional information on the data used (Sauer et al, 2009(Sauer et al, , 2011(Sauer et al, , 2014.…”

“…Moreover, it was not possible for the respondents to skip vignettes. The problem is described in greater detail in Sauer et al (2009Sauer et al ( , 2014. Thus, interviews with less than 5 min of processing time for the vignette module (less than 12 s of processing time per vignette) were discarded from the analysis sample.…”

“…The quality of the data from population sample 2 is therefore not as high as it is in the other two samples. Further details on the methodical setup of the factorial survey can be found in Sauer et al (2009Sauer et al ( , 2011Sauer et al ( , 2014. Note that the data from population sample 1 were used for the analysis published in Auspurg et al (2017) with a different approach and focus.…”

Legitimate authorities and rational taxpayers: An investigation of voluntary compliance and method effects in a survey experiment of income tax evasion

“…This suggests that order effects-if present-will be more pronounced in survey experiments with a greater number of (a) dimensions, (b) evaluation tasks, and (c) evaluated vignettes (see Auspurg and Jäckle, 2017). 7 Complexity effects, on the other hand, can be defined as the inconsistency of responses to evaluation tasks that emerge when respondents cope with a high amount of information (Auspurg et al, 2009;Sauer et al, 2011Sauer et al, , 2014. Complexity effects primarily stem from one key source: the number of dimensions manipulated in a vignette.…”

“…Anything related to the design of a survey experiment that places extra cognitive burden or fatigue on respondents should amplify complexity effects. This suggests that complexity effects-if present-will be more pronounced in survey experiments with a greater number of vignettes and a greater number of evaluation tasks presented to respondents (see Auspurg et al, 2009;Sauer et al, 2011Sauer et al, , 2014. To state it differently, both the number of vignettes and the number of evaluation tasks increase cognitive fatigue, and should worsen the bias introduced by increasing the number of dimensions manipulated in a vignette.…”