New treatments compared to established treatments in randomized trials

Abstract

Background The proportion of proposed new treatments that are ’successful’ is of ethical, scientific, and public importance. We investigated how often new, experimental treatments evaluated in randomized controlled trials (RCTs) are superior to established treatments. Objectives Our main question was: “On average how often are new treatments more effective, equally effective or less effective than established treatments?” Additionally, we wanted to explain the observed results, i.e. whether the observed distribution of outcomes is consistent with the ’uncertainty requirement’ for enrollment in RCTs. We also investigated the effect of choice of comparator (active versus no treatment/placebo) on the observed results. Search methods We searched the Cochrane Methodology Register (CMR) 2010, Issue 1 in The Cochrane Library (searched 31 March 2010); MEDLINE Ovid 1950 to March Week 2 2010 (searched 24 March 2010); and EMBASE Ovid 1980 to 2010 Week 11 (searched 24 March 2010). Selection criteria Cohorts of studies were eligible for the analysis if they met all of the following criteria: (i) consecutive series of RCTs, (ii) registered at or before study onset, and (iii) compared new against established treatments in humans. Data collection and analysis RCTs from four cohorts of RCTs met all inclusion criteria and provided data from 743 RCTs involving 297,744 patients. All four cohorts consisted of publicly funded trials. Two cohorts involved evaluations of new treatments in cancer, one in neurological disorders, and one for mixed types of diseases. We employed kernel density estimation, meta-analysis and meta-regression to assess the probability of new treatments being superior to established treatments in their effect on primary outcomes and overall survival. Main results The distribution of effects seen was generally symmetrical in the size of difference between new versus established treatments. Meta-analytic pooling indicated that, on average, new treatments were slightly more favorable both in terms of their effect on reducing the primary outcomes (hazard ratio (HR)/odds ratio (OR) 0.91, 99% confidence interval (CI) 0.88 to 0.95) and improving overall survival (HR 0.95, 99% CI 0.92 to 0.98). No heterogeneity was observed in the analysis based on primary outcomes or overall survival (I2 = 0%). Kernel density analysis was consistent with the meta-analysis, but showed a fairly symmetrical distribution of new versus established treatments indicating unpredictability in the results. This was consistent with the interpretation that new treatments are only slightly superior to established treatments when tested in RCTs. Additionally, meta-regression demonstrated that results have remained stable over time and that the success rate of new treatments has not changed over the last half century of clinical trials. The results were not significantly affected by the choice of comparator (active versus placebo/no therapy). Authors’ conclusions Society can expect that slightly more than half of new experimental treatm...

Meta-analysis of aggregate data on medical events

“…In fact, this prior puts more probability in the tail area than the distribution of the estimated hazard ratios for new drugs versus controls from several sets of historical trials (Djulbegovic et al, 2012) shown in sizes, but also of sampling variation. For this reason priors developed on this basis have been criticized for depending on the sampling variation in historical trials (Senn, 2007).…”

Clinical trial metadata: defining and extracting metadata on the design, conduct, results and costs of 125 randomised clinical trials funded by the National Institute for Health Research Health Technology Assessment programme

“…The recent Cochrane Review used data extracted for this project and combined them with the only three other similar cohorts. 96 …”

“…We have also recommended that the database be extended in relation to its inclusion of sufficient data on differences in results between new and old interventions, to enable it to remain part of the portfolio of cohorts of trials included in the ongoing Cochrane Review. 96 We recommend that the database be made available to other researchers, subject to similar confidentiality agreements that governed the present project (similar because the rules governing data access have changed since this project began). We also recommend that the database include aspects aimed at enabling the quality of its data to be improved by such use.…”

“…96 As two of us (LS, JR) had extracted and classified data on 65 published HTA programme trials, this would have been a considerable but not major challenge. Given that those 65 trials made up one of only four such cohorts globally, it is unfortunate that all 125 trials in the database cannot constitute an update on that earlier work.…”

“…5 This work led to inclusion of those trials as one of four cohorts in an international review of old versus new interventions in trials. 96 (This Cochrane Review showed that it identified four such cohorts globally, one based on US NIH cancer trials, one on 30 UK MRC trials and another on a cohort of 30 Canadian trials. The fourth cohort was made up of some 60 HTA-funded trials based on the previous work by JR and LS.)…”