Statistical challenges in assessing potential efficacy of complex interventions in pilot or feasibility studies

Abstract

Early phase trials of complex interventions currently focus on assessing the feasibility of a confirmatory RCT and on conducting pilot work. These trials are not designed to enable a formal assessment of potential efficacy. As a result, guidance recommends any statistical analysis of treatment effects conducted in these trials should be treated with extreme caution and not used in deciding if a confirmatory trial of the intervention is warranted. Phase II trial designs developed in the drug context offer a potential solution, providing methods for selecting sample size parameters for feasibility and pilot trials which will enable formal assessments of potential efficacy to be carried out. In this paper we will outline the challenges encountered in extending ideas developed in the phase II drug trial literature to the complex intervention setting. The prevalence of multiple endpoints and clustering of outcome data will be identified as important considerations, having implications for timely and robust determination of optimal sample size parameters. The potential for Bayesian methods to help to identify robust trial designs and optimal decision rules will also be explored.

Efficient and flexible simulation-based sample size determination for clinical trials with multiple design parameters

“…SSD problems, particularly those found in trials of complex interventions, are not always this simple. 13 There may be several dimensions to the trial’s sample size or, more generally, several quantitative parameters, which must be specified at the design stage and which influence the power of the trial. We will refer to these as design parameters .…”

Bayesian design and analysis of external pilot trials for complex interventions

“…The computational difficulties will be particularly pertinent when using our approach to determine sample size, as several evaluations of different sample size choices will be required. If the choice of sample size can be framed as an optimization problem, methods for efficient global optimization of computationally expensive functions such as those described by Jones 36 and Roustant et al 37 may be useful 16 . Alternatively, one of several rules‐of‐thumb for choosing pilot sample size 3,9,11,13 could be used, with the resulting operating characteristics evaluated using the proposed method.…”

“…While some have noted that the low sample size in pilots may lead to a considerable probability that a certain progression criterion will be met (or missed) due to random sampling variation, 12,15 and despite the consequences of making the wrong progression decision, the statistical properties of pilot decision rules are rarely used to inform the choice of sample size. This may be due to the methodological challenges commonly found in pilot trials of complex interventions, including the simultaneous evaluation of multiple endpoints, complex multi‐level models, small sample sizes, and prior uncertainty in nuisance parameters 16 …”

Determining sample size for progression criteria for pragmatic pilot RCTs: The Hypothesis test Strikes Back!

“…Horne et al, (21) reviewed 31 pilot trials published in physical therapy journals between 2012-5 and found that only 4/31 (13%) carried out a valid sample size calculation on effectiveness/e cacy outcomes but 26/31 (84%) used hypothesis testing. Wilson et al (22) acknowledged a number of statistical challenges in assessing potential e cacy of complex interventions in pilot and feasibility studies. The CONSORT extension (Eldridge et al, 2016) re-a rmed many researchers' views that formal hypothesis testing for effectiveness/e cacy is not recommended in pilot/feasibility studies since they are underpowered to do so.…”