Vignette studies use short descriptions of situations or persons (vignettes) that are usually shown to respondents within surveys in order to elicit their judgments about these scenarios. By systematically varying the levels of theoretically important vignette characteristics a large population of different vignettes is typically available – too large to be presented to each respondent. Therefore, each respondent gets only a subset of vignettes. These subsets may either be randomly selected in following the tradition of the factorial survey or systematically selected according to an experimental design. We show that these strategies in selecting vignette sets have strong implications for the analysis and interpretation of vignette data. Random selection strategies result in a random confounding of effects and heavily rely on the assumption of no interaction effects. In contrast, experimental strategies systematically confound interaction effects with main or set effects, thereby preserving a meaningful interpretation of main and important interaction effects. Using a pilot study on attitudes toward immigrants we demonstrate the implementation and analysis of a confounded factorial design.
The assumption of strongly ignorable treatment assignment is required for eliminating selection bias in observational studies. To meet this assumption, researchers often rely on a strategy of selecting covariates that they think will control for selection bias. Theory indicates that the most important covariates are those highly correlated with both the real selection process and the potential outcomes. However, when planning a study, it is rarely possible to identify such covariates with certainty. In this article, we report on an extensive reanalysis of a within-study comparison that contrasts a randomized experiment and a quasi-experiment. Various covariate sets were used to adjust for initial group differences in the quasi-experiment that was characterized by self-selection into treatment. The adjusted effect sizes were then compared with the experimental ones to identify which individual covariates, and which conceptually grouped sets of covariates, were responsible for the high degree of bias reduction achieved in the adjusted quasi-experiment. Such results provide strong clues about preferred strategies for identifying the covariates most likely to reduce bias when planning a study and when the true selection process is not known.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.