The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials

Abstract

Objective To summarise comparisons of randomised clinical trials and non-randomised clinical trials, trials with adequately concealed random allocation versus inadequately concealed random allocation, and high quality trials versus low quality trials where the effect of randomisation could not be separated from the effects of other methodological manoeuvres. Design Systematic review. Selection criteria Cohorts or meta-analyses of clinical trials that included an empirical assessment of the relation between randomisation and estimates of effect. Data sources Cochrane Review Methodology Database, Medline, SciSearch, bibliographies, hand searching of journals, personal communication with methodologists, and the reference lists of relevant articles. Main outcome measures Relation between randomisation and estimates of effect. Results Eleven studies that compared randomised controlled trials with non-randomised controlled trials (eight for evaluations of the same intervention and three across different interventions), two studies that compared trials with adequately concealed random allocation and inadequately concealed random allocation, and five studies that assessed the relation between quality scores and estimates of treatment effects, were identified. Failure to use random allocation and concealment of allocation were associated with relative increases in estimates of effects of 150% or more, relative decreases of up to 90%, inversion of the estimated effect and, in some cases, no difference. On average, failure to use randomisation or adequate concealment of allocation resulted in larger estimates of effect due to a poorer prognosis in non-randomly selected control groups compared with randomly selected control groups. Conclusions Failure to use adequately concealed random allocation can distort the apparent effects of care in either direction, causing the effects to seem either larger or smaller than they really are. The size of these distortions can be as large as or larger than the size of the effects that are to be detected.

Are there roles for observational database studies and structured quantification of expert opinion to answer therapy controversies in transplants?

“…The size of these distortions can be as large as or larger than the size of the effects that are to be detected. 9 Use of biological assignment rather than randomization, as is sometimes done in transplant studies, also raises potential for enrollment bias to impact study results. 10 As we discussed above, most physicians want a generic rather than specific answer to the question whether autotransplants are better than chemotherapy in persons with AML in first remission.…”

Effects of Long-Term Low-Molecular-Weight Heparin on Fractures and Bone Density in Non-Pregnant Adults: A Systematic Review With Meta-Analysis

“…However, the observational studies examined bone outcomes as their primary endpoints. Although selection bias and confounding can be better controlled through allocation concealment and randomization in RCTs, 72 the majority of the reviewed trials examined BMD and/or fractures as secondary end points and were adequately powered for their primary efficacy end points. In this review, BMD was the primary outcome in one cross-over trial only.…”

“…[35][36][37] Given the various methodological limitations of the cohort studies (selection bias, confounding, small samples, inadequate ascertainment of outcomes), the effect of LMWH on BMD can be distorted in either direction. 72, 73 Thus, we qualitatively synthesized the BMD Figure 2. a Forest plot: The effect of long-term low-molecularweight heparin vs. control on all fractures in non-pregnant participants.…”

The evolution of the randomized controlled trial and its role in evidence‐based decision making

“…Although there is some debate about the order within the hierarchies there has been a profound shift to move RCTs and systematic reviews of RCTs to the top of the therapeutic hierarchy. The placement of RCTs at the top of the therapeutic research hierarchy has occurred due to the realization that RCTs are superior to observational studies in evaluating treatments because RCTs eliminate bias in the choice of treatment assignments and provide the only means to control for unknown prognostic factors [30]. Further there have been several examples where there have been contradictory findings between observational studies and RCTs (Table 4).…”