Sensitivity analyses for effect modifiers not observed in the target population when generalizing treatment effects from a randomized controlled trial: Assumptions, models, effect scales, data scenarios, and implementation details

Abstract

BackgroundRandomized controlled trials are often used to inform policy and practice for broad populations. The average treatment effect (ATE) for a target population, however, may be different from the ATE observed in a trial if there are effect modifiers whose distribution in the target population is different that from that in the trial. Methods exist to use trial data to estimate the target population ATE, provided the distributions of treatment effect modifiers are observed in both the trial and target population—an assumption that may not hold in practice.MethodsThe proposed sensitivity analyses address the situation where a treatment effect modifier is observed in the trial but not the target population. These methods are based on an outcome model or the combination of such a model and weighting adjustment for observed differences between the trial sample and target population. They accommodate several types of outcome models: linear models (including single time outcome and pre- and post-treatment outcomes) for additive effects, and models with log or logit link for multiplicative effects. We clarify the methods’ assumptions and provide detailed implementation instructions.IllustrationWe illustrate the methods using an example generalizing the effects of an HIV treatment regimen from a randomized trial to a relevant target population.ConclusionThese methods allow researchers and decision-makers to have more appropriate confidence when drawing conclusions about target population effects.

Extending inferences from a randomized trial to a target population

“…These judgments can be informed by examining readily available data on the variation of treatment effects among subgroups defined in terms of observed variables in the data at hand or external sources, including observational studies; the variation of treatment effects across studies examining similar interventions and outcomes in different populations (e.g., as assessed in meta-analyses); or the variation of the mean outcome under each treatments across populations and population subgroups. The benefit of our approach becomes clear when compared against approaches that require the specification of models for the distribution of unmeasured variables and the associations between unmeasured and measured variables (e.g., [11,12]). These alternative approaches have multiple sensitivity parameters and require detailed background knowledge about sources of effect heterogeneity; such knowledge is often unavailable because empirical studies typically do not allow the precise assessment of effect modification [34,35].…”

“…The literature on sensitivity analysis for unmeasured confounding in observational studies or for the related problem of data missing not-at-random is very extensive (starting with [7] and expanded in various ways, e.g., [8][9][10]). In the context of analyses extending inferences from a randomized trial to a target population, however, the only proposal for sensitivity analysis methods that we are aware of is the work of Nguyen et al [11,12]. Their approach can be useful when background knowledge is strong enough to suggest that a single variable that would render randomized and non-randomized groups exchangeable has been measured among the former but not among the latter.…”

Extending inferences from a randomized trial to a new target population

“…Because the mean exchangeability assumption is not testable, it is reasonable to explore how conclusions would change under different degrees of assumption violations. Depending on the available data and background knowledge, various sensitivity analysis methods for missing data or unmeasured confounding can be modified for use in analyses extending inferences to a target population . In a companion paper, we have proposed easy to implement estimators for sensitivity analysis that are based on the estimators described in Section 5 and do not require extensive background knowledge about the distribution of unknown or unmeasured variables that are the source of violations of the exchangeability condition …”

Transporting experimental results with entropy balancing

“…32,33 Additional sensitivity analyses should be employed intermittently to ensure that every effect modifier is accounted for. 34 In the case of positivity violations, it seems that methods that employ more direct implementations of an outcome model, such as the G-computation and DR approaches, far better given their ability to extrapolate over the covariate space. Violations of Assumptions 1 and 3 pose a more difficult challenge to evaluate as these assumptions are untestable.…”