Generalizability of findings from randomized controlled trials: application to the National Institute of Drug Abuse Clinical Trials Network

Abstract

Aims To compare randomized controlled trial (RCT) sample treatment effects with the population effects of substance use disorder (SUD) treatment. Design Statistical weighting was used to re-compute the effects from ten RCTs such that the participants in the trials had characteristics that resembled those of patients in the target populations. Settings Multi-site RCTs and usual SUD treatment settings in the USA. Participants A total of 3,592 patients in ten RCTs and 1,602,226 patients from usual SUD treatment settings between 2001 and 2009. Measurements Three outcomes of SUD treatment were examined: retention, urine toxicology, and abstinence. We weighted the RCT sample treatment effects using propensity scores representing the conditional probability of participating in RCTs. Findings Weighting the samples changed the significance of estimated sample treatment effects. Most commonly, positive effects of trials became statistically non-significant after weighting (three trials for retention and urine toxicology, and one trial for abstinence); but also, non-significant effects became significantly positive (one trial for abstinence), and significantly negative effects became non-significant (two trials for abstinence). There was suggestive evidence of treatment effect heterogeneity in subgroups that are under- or over-represented in the trials, some of which were consistent with the differences in average treatment effects between weighted and unweighted results. Conclusions The findings of randomized controlled trials (RCTs) for substance use disorder treatment do not appear to be directly generalizable to target populations when the RCT samples do not adequately reflect the target populations and there is treatment effect heterogeneity across patient subgroups.

Effect heterogeneity and variable selection for standardizing causal effects to a target population

“…Inverse probability weighted methods have been applied to generalize the results of trials on the effect of HIV medication [28] and treatments for substance use disorder [29]. Lesko et al provided a full description of how these methods can be used in practice [20].…”

Opioid agonist treatment and fatal overdose risk in a state‐wide US population receiving opioid use disorder services

“…The effectiveness of medication versus non-medication treatment in usual care contexts could differ from clinical trials, where patients are known to be characteristically different than those receiving care in real-world settings [21]. Key populations may be excluded from clinical trials (e.g.…”

The effectiveness and safety of nonsurgical integrative interventions for symptomatic lumbar spinal spondylolisthesis

“…An randomized controlled trial, which can reduce the risk of bias in study participant selection, may not be able to ensure external validity if the study population is limited to homogeneous group under strict eligibility criteria. [ 21 ] From this perspective, a multinational, multicenter clinical trial has benefits of ensuring generalizability because patients with diverse regional and ethnic characteristics are recruited.…”