Strengthening causal inference from randomised controlled trials of complex interventions

Abstract

Researchers conducting randomised controlled trials (RCTs) of complex interventions face design and analytical challenges that are not fully addressed in existing guidelines. Further guidance is needed to help ensure that these trials of complex interventions are conducted to the highest scientific standards while maximising the evidence that can be extracted from each trial. The key challenge is how to manage the multiplicity of outcomes required for the trial while minimising false positive and false negative findings. To address this challenge, we formulate three principles to conduct RCTs: (1) outcomes chosen should be driven by the intent and programme theory of the intervention and should thus be linked to testable hypotheses; (2) outcomes should be adequately powered and (3) researchers must be explicit and fully transparent about all outcomes and hypotheses before the trial is started and when the results are reported. Multiplicity in trials of complex interventions should be managed through careful planning and interpretation rather than through post hoc analytical adjustment. For trials of complex interventions, the distinction between primary and secondary outcomes as defined in current guidelines does not adequately protect against false positive and negative findings. Primary outcomes should be defined as outcomes that are relevant based on the intervention intent and programme theory, declared (ie, registered), and adequately powered. The possibility of confirmatory causal inference is limited to these outcomes. All other outcomes (either undeclared and/or inadequately powered) are secondary and inference relative to these outcomes will be exploratory.

Prevalence of morbidity symptoms among pregnant and postpartum women receiving different nutrient supplements in Ghana and Malawi: A secondary outcome analysis of two randomised controlled trials

“…counterproductive (Streiner, 2015) or inappropriate (Leroy et al, 2022;Rothman, 1990;Schulz & Grimes, 2005).…”

“…Third, we tested multiple hypotheses involving 11 or 10 morbidity symptoms in either country and therefore, at least some of the observed differences may be due to chance (Li et al, 2017 ). We did not correct for multiplicity in our hypotheses testing because we considered any such correction as counterproductive (Streiner, 2015 ) or inappropriate (Leroy et al, 2022 ; Rothman, 1990 ; Schulz & Grimes, 2005 ).…”

Effects of engaging fathers and bundling parenting and nutrition interventions on early child development and maternal and paternal parenting in Mara, Tanzania: a factorial cluster‐randomized controlled trial

“…In particular, psychometric evidence regarding the performance, optimal scoring approach, and stability in scores over time for the Bayley Scales remains unclear in the Tanzanian context, as seen in our varied results upon using Bayley composite scores versus internally standardized scores. We conducted many statistical tests without adjustment for multiple comparisons (Feise, 2002; Leroy et al., 2022; Rothman, 1990). This increases the potential risk of observing significant findings based on a random sampling error.…”

“…For continuous outcomes, we report results as standardized coefficients for comparability and ease of interpretation. We did not adjust for multiple comparisons as several studies have argued that such corrections can be inappropriate by increasing type II error (i.e., not detecting an impact when it exists) and lead to inaccurate inferences, especially when applied post hoc, and not all outcomes were powered by such analyses (Feise, 2002; Leroy et al., 2022; Rothman, 1990). Analyses were conducted in Stata version 15.0.…”

Exploring human biology with N-of-1 clinical trials

“…then a good health intervention should at a minimum not negatively affect any of them and at best positively affect them all. In this light, testing multiple measures for intervention effects simultaneously using an omnibus statistical test of the hypothesis that an intervention positively effects them all could lead to an increase in power (Huitema, 2011 ; Tabachnick and Fidell, 2012 ), but only if the number of measures is large (Leroy et al, 2022 ). Reaching appropriate numbers of observations could be achieved, for example, through the use of the aforementioned continuous wireless devices or microsampling techniques which involve collecting minute amounts of blood or urine for analyses to avoid a standard blood draw or logistically challenging biospecimen collections (Enderle et al, 2016 ; Bentley et al, 2019 ; Anderson et al, 2020 ).…”