Understanding and misunderstanding randomized controlled trials

Abstract

Randomized Controlled Trials (RCTs) are increasingly popular in the social sciences, not only in medicine. We argue that the lay public, and sometimes researchers, put too much trust in RCTs over other methods of investigation. Contrary to frequent claims in the applied literature, randomization does not equalize everything other than the treatment in the treatment and control groups, it does not automatically deliver a precise estimate of the average treatment effect (ATE), and it does not relieve us of the need to think about (observed or unobserved) covariates. Finding out whether an estimate was generated by chance is more difficult than commonly believed. At best, an RCT yields an unbiased estimate, but this property is of limited practical value. Even then, estimates apply only to the sample selected for the trial, often no more than a convenience sample, and justification is required to extend the results to other groups, including any population to which the trial sample belongs, or to any individual, including an individual in the trial. Demanding ‘external validity’ is unhelpful because it expects too much of an RCT while undervaluing its potential contribution. RCTs do indeed require minimal assumptions and can operate with little prior knowledge. This is an advantage when persuading distrustful audiences, but it is a disadvantage for cumulative scientific progress, where prior knowledge should be built upon, not discarded. RCTs can play a role in building scientific knowledge and useful predictions but they can only do so as part of a cumulative program, combining with other methods, including conceptual and theoretical development, to discover not ‘what works’, but ‘why things work’.

Withdrawal Confounding in Randomized Controlled Trials of Antipsychotic, Antidepressant, and Stimulant Drugs, 2000–2017

“…The current study’s focus on a potential withdrawal confound adds to existing conceptual critiques of the RCT design [18], including that participants who complete an RCT may differ from the larger group of participants at the beginning of the trial [18, 32], and that the “placebo model” in RCTs fails to account for the combined effects of several other environmental, psychological, and social variables at work in any trial [33]. Online suppl.…”

“…Since then, possible withdrawal bias has been identified and discussed [3, 13], with suggestions that it inflates relapse rates and overstates drugs’ effectiveness [14-16]. Critiques of psychotropic drug RCTs [17, 18] and standard risk of bias assessment tools in clinical trials [19] do not mention withdrawal confound as a potential for bias. A companion paper, a recent systematic review of the employment of the discontinuation procedure in RCTs of all psychotropic drugs, sees little attention to withdrawal confound in relapse prevention trials [20].…”

All that Glitters is not Gold. The Political Economy of Randomized Evaluations in Development

“…Yet RCTs have a number of practical and theoretical limitations. The main theoretical criticism concerns the tension between bias and precision, where RCTs fail to make an optimal trade-off (Barrett and Carter, 2010;Deaton, 2010;Deaton and Cartwright, 2016;Heckman, 1991;Ravallion, 2009;Rodrik, 2008). Randomized analyses focus on average results for the entire population being considered.…”

“…The restriction to a short-term impact (for reasons of cost and attrition) often means that mid-point indicators are studied, which can be very different from final outcomes (Eble et al, 2014), if not the reverse, since many project trajectories are not linear (Labrousse, 2010;Woolcock, 2013). Knock-on and general equilibrium effects are ignored despite there being any number of them (Acemoglu, 2010;Deaton and Cartwright, 2016;Ravallion, 2009). The same holds true for the political aspect of programme replication, despite its being a key consideration for scale-up (Acemoglu, 2010;Bold et al, 2013;Pritchett and Sandefur, 2013).…”

Perioperative goal-directed therapy: what’s the best study design to investigate its impact on patient outcome?

“…However, high-quality pragmatic randomized controlled trials are missing for many anesthesiology treatment strategies. In addition, as discussed previously, in heterogeneous patient populations-such as "surgical patients"-with complex multifactorial outcomes, randomized controlled trials and their meta-analyses have their own limitations and need to be meticulously designed [7,[11][12][13].…”