Generalizing from Unrepresentative Experiments: A Stratified Propensity Score Approach

Abstract

The paper addresses means of generalizing from an experiment based on a nonprobability sample to a population of interest and to subpopulations of interest, where information is available about relevant covariates in the whole population. Using stratification based on propensity score matching with an external populationwide data set, an estimator of the population average treatment effect is constructed. An example is presented in which the applicability of a major education intervention in a non-probability sample of schools in Texas, USA, is assessed for the state as a whole and for its constituent counties. The implications of the results are discussed for two important situations: how to use this methodology to establish where future experiments should be conducted to improve this generalization and how to construct a priori a strategy for experimentation which will maximize both the initial inferential power and the final inferential basis for a series of experiments.

Recruiting the ABCD sample: Design considerations and procedures

“…There is a growing appreciation that, rather than being restricted to convenience samples that often exclude harder-to-recruit socio-economic groups, research studies should strive to increase the diversity of their samples (Keiding and Louis, 2016). Consistent with this aim of achieving greater “external validity” for both experimental and observational study findings, there is a growing body of statistical literature and methods that aim to maximize the population representativeness even when study conditions do not permit a full application of traditional probability sampling methods (see Dugoff et al 2014; O’Muircheartaigh and Hodges, 2014; Stuart et al, 2015; Elliott and Valliant, 2017). While there now exists institutional commitment to ensuring sex differences are incorporated into study designs (https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15–102.html) the same is not yet true for other factors such as race and ethnicity, SES and urbanicity.…”

Assessing the Generalizability of Randomized Trial Results to Target Populations

“…In addition, the topic of external validity has not been discussed as much in the social and behavioral sciences (with a few recent exceptions, including Olsen et al, 2013; O’Muircheartaigh & Hedges, 2014; Tipton, 2013). Some of the considerations may be quite different in different fields.…”

Estimation of Population Average Treatment Effects in the FIRST Trial: Application of a Propensity Score‐Based Stratification Approach

“…After a study has concluded, generalizability can still be assessed through the use of propensity score‐based subclassification approaches such as that developed by O'Muircheartaigh and Hedges () and as we demonstrated in this paper. An advantage of this approach is that it only requires data sufficient to estimate a propensity model of study participation.…”

“…We demonstrate how this question can be addressed using a propensity score‐based poststratification approach developed by O'Muircheartaigh and Hedges () to estimate population average treatment effects from studies involving nonrepresentative samples. We apply this method to data from the Flexibility in Duty Hour Requirements for Surgical Trainees Trial (“FIRST Trial”) to estimate the average effect of a resident duty hour policy change in the population of general surgery residency programs and affiliated hospitals (of which the FIRST Trial sample was a subset) and the subset of the inference population that did not participate in the FIRST Trial.…”