Information Equivalence in Survey Experiments

Abstract

Survey experiments often manipulate the description of attributes in a hypothetical scenario, with the goal of learning about those attributes’ real-world effects. Such inferences rely on an underappreciated assumption: experimental conditions must be information equivalent (IE) with respect to background features of the scenario. IE is often violated because subjects, when presented with information about one attribute, update their beliefs about others too. Labeling a country “a democracy,” for example, affects subjects’ beliefs about the country’s geographic location. When IE is violated, the effect of the manipulation need not correspond to the quantity of interest (the effect of beliefs about the focal attribute). We formally define the IE assumption, relating it to the exclusion restriction in instrumental-variable analysis. We show how to predict IE violations ex ante and diagnose them ex post with placebo tests. We evaluate three strategies for achieving IE. Abstract encouragement is ineffective. Specifying background details reduces imbalance on the specified details and highly correlated details, but not others. Embedding a natural experiment in the scenario can reduce imbalance on all background beliefs, but raises other issues. We illustrate with four survey experiments, focusing on an extension of a prominent study of the democratic peace.

Authoritarian Audiences, Rhetoric, and Propaganda in International Crises: Evidence from China

“…As hypothetical scenarios become more abstract and devoid of contextual information, the connection between survey responses and real-world reactions to particular crises becomes more tenuous, weakening external validity. Conversely, respondents may think of real world examples in answering questions about abstract scenarios, introducing a form of bias akin to confounding biases in observational studies (Dafoe et al, 2018). For example, Chinese respondents who read that an unnamed "neighboring country" is a powerful democracy and a US ally are more likely to think of Japan, plausibly influencing their responses in unintended ways.…”

The Policy Basis of Measured Partisan Animosity in the United States

“…In addition to allowing us to understand the robustness and underlying nature of partisan divisions, this approach enables us to measure the depth and nature of divisions on the basis of issue positions. More generally, it allows us to test a crucial assumption about the interpretation of survey experiments, which is whether the manipulations operate through the hypothesized mechanism or other potential causal pathways (a violation of the exclusion restriction; see Dafoe, Zhang, and Caughey ). Finally, in terms of external validity, we argue that survey items that provide a greater breadth of information more accurately capture private social environments in which individuals evaluate others.…”

(Un)principled principals, (un)principled agents: The differential effects of managerial civil service reforms on corruption in developing and OECD countries

“…Following Oliveros and Schuster (2018), our conjoint also includes three common correlates of corruption: education (university versus high school), rank (administrative support vs. technical-professional), and years of experience in the public sector (year of entry in 2005, 2010, or 2015). The additional attributes strengthen the realism of the presented profiles and reduce confounding risks (see Dafoe, Zhang, & Caughey, 2018).…”