No abstract

Randomized experiments are considered the gold standard for causal inference, as they can provide unbiased estimates of treatment effects for the experimental participants. However, researchers and policymakers are often interested in using a specific experiment to inform decisions about other target populations. In education research, increasing attention is being paid to the potential lack of generalizability of randomized experiments, as the experimental participants may be unrepresentative of the target population of interest. This paper examines whether generalization may be assisted by statistical methods that adjust for observed differences between the experimental participants and members of a target population. The methods examined include approaches that reweight the experimental data so that participants more closely resemble the target population and methods that utilize models of the outcome. Two simulation studies and one empirical analysis investigate and compare the methods’ performance. One simulation uses purely simulated data while the other utilizes data from an evaluation of a school-based dropout prevention program. Our simulations suggest that machine learning methods outperform regression-based methods when the required structural (ignorability) assumptions are satisfied. When these assumptions are violated, all of the methods examined perform poorly. Our empirical analysis uses data from a multi-site experiment to assess how well results from a given site predict impacts in other sites. Using a variety of extrapolation methods, predicted effects for each site are compared to actual benchmarks. Flexible modeling approaches perform best, although linear regression is not far behind. Taken together, these results suggest that flexible modeling techniques can aid generalization while underscoring the fact that even state-of-the-art statistical techniques still rely on strong assumptions.

In this case study of the impact of West German television on public support for the East German communist regime, we evaluate the conventional wisdom in the democratization literature that foreign mass media undermine authoritarian rule. We exploit formerly classified survey data and a natural experiment to identify the effect of foreign media exposure using instrumental variable estimators. Contrary to conventional wisdom, East Germans exposed to West German television were more satisfied with life in East Germany and more supportive of the East German regime. To explain this surprising finding, we show that East Germans used West German television primarily as a source of entertainment. Behavioral data on regional patterns in exit visa applications and archival evidence on the reaction of the East German regime to the availability of West German television corroborate this result.

Regression discontinuity (RD) designs enable researchers to estimate causal effects using observational data. These causal effects are identified at the point of discontinuity that distinguishes those observations that do or do not receive the treatment. One challenge in applying RD in practice is that data may be sparse in the immediate vicinity of the discontinuity. Expanding the analysis to observations outside this immediate vicinity may improve the statistical precision with which treatment effects are estimated, but including more distant observations also increases the risk of bias. Model specification is another source of uncertainty; as the bandwidth around the cutoff point expands, linear approximations may break down, requiring more flexible functional forms. Using data from a large randomized experiment conducted by Gerber, Green, and Larimer (2008), this study attempts to recover an experimental benchmark using RD and assesses the uncertainty introduced by various aspects of model and bandwidth selection. More generally, we demonstrate how experimental benchmarks can be used to gauge and improve the reliability of RD analyses.

No abstract

Do foreign media facilitate the diffusion of protest in authoritarian regimes? Apparently for the first time, the author tests this hypothesis using aggregate and survey data from communist East Germany. The aggregate-level analysis takes advantage of the fact that West German television broadcasts could be received in most but not all parts of East Germany. The author exploits this “natural experiment” by conducting a matched analysis in which counties without West German television are matched to a comparison group of counties with West German television. Comparing these two groups of East German counties, the author finds no evidence that West German television affected the speed or depth of protest diffusion during the 1989 East German revolution. He also analyzes a survey of East German college students. Confirming the aggregate-level results, the survey data show that, at least among college students, exposure to West German television did not increase protest participation.

No abstract

Although public administration scholars have long studied discrimination on the basis of race/ethnicity, class, and gender, little to no research exists on whether street‐level bureaucrats provide differential services based on the religious identity of their constituents. This article reports the results from a large‐scale correspondence study of street‐level bureaucrats in the American public school system. The authors emailed the principals of a large sample of public schools and asked for a meeting, randomly assigning the religious (non)affiliation of the family. To get at potential causal mechanisms, religious belief intensity was also randomly assigned. The findings show evidence of substantial discrimination against Muslims and atheists on a par with, and sometimes larger than, the racial discrimination found in previous studies. These individuals are substantially less likely to receive a response, with discrimination growing when they signal that their beliefs are more intense. Protestants and Catholics face no discrimination unless they signal that their religious beliefs are intense.