Increasing Precision in Survey Experiments Without Introducing Bias

Abstract

The use of survey experiments has surged in political science as a method for estimating causal effects. By far, the most common design is the betweensubjects design in which the outcome is only measured posttreatment. This design relies heavily on recruiting a large number of subjects to achieve adequate statistical power. Alternative designs that involve repeated measurement of the dependent variable promise greater precision, but are rarely used out of fears that these designs will bias treatment effects (e.g., due to consistency pressures). Across six studies, we assess this conventional wisdom by testing experimental designs against each other. Our results demonstrate that repeated measures designs substantially increase precision, while introducing little to no bias. These designs also offer new insights into the nature of treatment effects. We conclude by encouraging researchers to adopt repeated measures designs and providing guidelines for when and how to use them.

Nudging Enforcers: How Norm Perceptions and Motives for Lying Shape Sanctions

“…We examine questions through the lens of two experiments: the first experiment 22 Recall that Experiment 1 is a within-subject design with respect to the different lying scenarios whereas Experiment 2 varies all treatment dimensions between subjects. The fact that results are nevertheless very consistent across these independent experiments suggests that our results are robust to both of these design choices (see also Clifford et al, 2020, for a recent discussion on the robustness of within-designs).…”

Nudging Enforcers: How Norm Perceptions and Motives for Lying Shape Sanctions

“…28 Recall that the behavioral experiment in Study 1 is a within-subjects design with respect to the different lying scenarios, whereas Study 2 varies all treatment dimensions between subjects. The fact that results are nevertheless very consistent across these independent experiments suggests that our results are robust to both of these design choices (see also Clifford et al, 2020, for a recent discussion on the robustness of within-designs).…”

Estimating the between-issue variation in party elite cue effects

“…While the risk of demand effects biasing treatment estimates appears low in designs like that which I use here (Clifford, Sheagley, and Piston 2020;Mummolo and Peterson 2019), to mitigate any risk of demand effects due to the within-subjects design, I…”