Overrunning in clinical trials: some thoughts from a methodological review

Stallard

et al. 2022

BMC Med Res Methodol

Background Assessing the long term effects of many surgical interventions tested in pragmatic RCTs may require extended periods of participant follow-up to assess effectiveness and use patient-reported outcomes that require large sample sizes. Consequently the RCTs are often perceived as being expensive and time-consuming, particularly if the results show the test intervention is not effective. Adaptive, and particularly group sequential, designs have great potential to improve the efficiency and cost of testing new and existing surgical interventions. As a means to assess the potential utility of group sequential designs, we re-analyse data from a number of recent high-profile RCTs and assess whether using such a design would have caused the trial to stop early. Methods Many pragmatic RCTs monitor participants at a number of occasions (e.g. at 6, 12 and 24 months after surgery) during follow-up as a means to assess recovery and also to keep participants engaged with the trial process. Conventionally one of the outcomes is selected as the primary (final) outcome, for clinical reasons, with others designated as either early or late outcomes. In such settings, novel group sequential designs that use data from not only the final outcome but also from early outcomes at interim analyses can be used to inform stopping decisions. We describe data from seven recent surgical RCTs (WAT, DRAFFT, WOLLF, FASHION, CSAW, FIXDT, TOPKAT), and outline possible group sequential designs that could plausibly have been proposed at the design stage. We then simulate how these group sequential designs could have proceeded, by using the observed data and dates to replicate how information could have accumulated and decisions been made for each RCT. Results The results of the simulated group sequential designs showed that for two of the RCTs it was highly likely that they would have stopped for futility at interim analyses, potentially saving considerable time (15 and 23 months) and costs and avoiding patients being exposed to interventions that were either ineffective or no better than standard care. We discuss the characteristics of RCTs that are important in order to use the methodology we describe, particularly the value of early outcomes and the window of opportunity when early stopping decisions can be made and how it is related to the length of recruitment period and follow-up. Conclusions The results for five of the RCTs tested showed that group sequential designs using early outcome data would have been feasible and likely to provide designs that were at least as efficient, and possibly more efficient, than the original fixed sample size designs. In general, the amount of information provided by the early outcomes was surprisingly large, due to the strength of correlations with the primary outcome. This suggests that the methods described here are likely to provide benefits more generally across the range of surgical trials and more widely in other application areas where trial designs, outcomes and follow-up patterns are structured and behave similarly.

Section: Methodsmentioning

confidence: 99%

Group sequential designs in pragmatic trials: feasibility and assessment of utility using data from a number of recent surgical RCTs

Stallard

et al. 2022

BMC Med Res Methodol

“…The second method is not only statistically insensible due to missing out potential for stronger evidence, but could also be related to ethical and regulatory issues since subjects have been treated but their data is not used afterwards whereas regulatory guidelines from the EMA demand the use of all patient data 12 . Furthermore, Baldi et al 13 demonstrated that additional incorporation of pipeline data can seriously impact the trial conclusion. In this paper, we will present methods making use of both the information

{I}_1

as well as the pipeline data in anticipation.…”

Section: The Test Problem and Delayed Response Modelmentioning

confidence: 99%

A two‐stage group‐sequential design for delayed treatment responses with the possibility of trial restart

Schüürhuis,

Konietschke,

Kunz

2024

Statistics in Medicine

Common statistical theory applicable to confirmatory phase III trial designs usually assumes that patients are enrolled simultaneously and there is no time gap between enrollment and outcome observation. However, in practice, patients are enrolled successively and there is a lag between the enrollment of a patient and the measurement of the primary outcome. For single‐stage designs, the difference between theory and practice only impacts on the trial duration but not on the statistical analysis and its interpretation. For designs with interim analyses, however, the number of patients already enrolled into the trial and the number of patients with available outcome measurements differ, which can cause issues regarding the statistical analyses of the data. The main issue is that current methodologies either imply that at the time of the interim analysis there are so‐called pipeline patients whose data are not used to make a statistical decision (like stopping early for efficacy) or the enrollment into the trial needs to be at least paused for interim analysis to avoid pipeline patients. There are methods for delayed responses available that introduced error‐spending stopping boundaries for the enrollment of patients followed by critical values to reject the null hypothesis in case the stopping boundaries have been crossed beforehand. Here, we will discuss other solutions, considering different boundary determination algorithms using conditional power and introducing a design allowing for recruitment restart while keeping the type I error rate controlled.

“…Also, at each interim analysis data on all those study participants recruited up to that point were used to estimated model parameters for the over-running analysis. 80,142 This analysis comprised all the data that would eventually have accumulated on those participants already recruited and is the definitive analysis that would have been reported, had the trial stopped recruiting at the interim analysis. In the over-running analysis, boundaries were recalculated based on the observed information at the final analysis and used to draw interference on the observed treatments effects.…”

Section: Information Monitoringmentioning

confidence: 99%

Subacromial spacers for adults with symptomatic, irreparable rotator cuff tears: the START:REACTS novel group sequential adaptive RCT

Metcalfe,

Arnold,

Parsons

et al. 2023

Efficacy Mech Eval

Background A balloon spacer is a relatively simple addition to an arthroscopic debridement procedure for irreparable rotator cuff tears. Objective To evaluate the clinical and cost-effectiveness of a subacromial balloon spacer for individuals undergoing arthroscopic debridement for irreparable rotator cuff tears. Design A multicentre participant-and assessor-blinded randomised controlled trial comparing arthroscopic debridement with the InSpace® (Stryker, Kalamazoo, MI, USA) balloon to arthroscopic debridement alone, using a novel adaptive design. Pretrial simulations informed stopping boundaries for two interim analyses, using outcome data from early and late time points. Setting A total of 24 NHS centres. Participants Adults with a symptomatic, irreparable rotator cuff tear for whom conservative management had been unsuccessful. Interventions Arthroscopic debridement of the subacromial space plus insertion of the InSpace balloon compared with arthroscopic debridement alone. Main outcome measures Oxford Shoulder Score at 12 months. Results A predefined stopping boundary was met at the first interim analysis. Recruitment stopped with 117 participants randomised. We obtained primary outcome data on 114 participants (97%). The mean Oxford Shoulder Score at 12 months was 34.3 in the debridement-only group (59 participants of 61 randomised) and 30.3 in the debridement with balloon group (55 participants of 56 randomised; mean difference: −4.2; 95% confidence interval −8.2 to −0.26; p = 0.037). There was no difference in safety events. In the cost-effectiveness analysis, debridement-only dominated with a probability of <1% that the device is cost-effective. Magnetic resonance imaging substudy To evaluate the function of the balloon, we developed a dynamic magnetic resonance imaging protocol to induce humeral movement by activating the deltoid. The pandemic restricted recruitment, so the sample size was small (n = 17). Statistical methodology study We applied the novel adaptive design approach to data from seven previous randomised controlled trials. The method would have been applicable to five of these trials and would have made substantial savings in time to recruitment, without compromising the main findings of the included trials. Interim analysis interpretation study We asked potential data monitoring committee members to review interim analysis reports presented using Bayesian and frequentist frameworks. They did not always follow the stopping rules and would benefit from additional information to support decision-making. Limitations The InSpace balloon could be beneficial in a different population although we are not aware of it being widely used for other purposes. As a result of the pandemic, we were not able to complete data collection for objective measures. Conclusions In this efficient adaptive trial, clinical and cost-effectiveness favoured the control treatment without the InSpace balloon. Therefore, we do not recommend this device for the treatment of irreparable rotator cuff tears. Future work There is an urgent need for high-quality research into interventions for people with irreparable rotator cuff tears as there is a lack of good evidence for all available treatment options at present. Trial registration This trial is registered as ISRCTN17825590. Funding This project (project reference 16/61/18) was funded by the Efficacy and Mechanism Evaluation (EME) Programme, a Medical Research Council and National Institute for Health and Care Research (NIHR) partnership. The trial is co-sponsored by the University of Warwick and University Hospitals Coventry and Warwickshire NHS Trust. This study will be published in full in Efficacy and Mechanism Evaluation; Vol. 10, No 3. See the NIHR Journals Library website for further project information.