Dropping subjects based on the results of a manipulation check following treatment assignment is common practice across the social sciences, presumably to restrict estimates to a subpopulation of subjects who understand the experimental prompt. We show that this practice can lead to serious bias and argue for a focus on what is revealed without discarding subjects. Generalizing results developed in Zhang and Rubin (2003) and Lee (2009) to the case of multiple treatments, we provide sharp bounds for potential outcomes among those who would pass a manipulation check regardless of treatment assignment. These bounds may have large or infinite width, implying that this inferential target is often out of reach. As an application, we replicate Press, Sagan, and Valentino (2013) with a design that does not drop subjects that failed the manipulation check and show that the findings are likely stronger than originally reported. We conclude with suggestions for practice, namely alterations to the experimental design.

No abstract

The Treaty on the Prohibition of Nuclear Weapons (TPNW) poses a challenge to decades of Japanese nuclear policy. While Japan has relied on the US nuclear umbrella since the aftermath of World War II, numerous pro-disarmament groups -including the Hibakushaare calling for Tokyo to join the Treaty. We contribute to these discussions with commentary on a new national survey we conducted in Japan (N = 1,333). Our results indicate that baseline support for the Prime Minister signing and the Diet ratifying the TPNW stands at approximately 75% of the Japanese public. Only 17.7% of the population is opposed, and 7.3% is undecided. Moreover, this support is cross-cutting, with a wide majority of every demographic group in the country favoring nuclear disarmament. Most strikingly, an embedded survey experiment demonstrates that the Japanese government cannot shift public opinion to oppose the Ban through the use of policy arguments or social pressure. Such broad support for the TPNW indicates that the Japanese government will not be able to hide from the Treaty and must take action to restore its credibility as a leader on nuclear disarmament.