Compared with what? An analysis of control‐group types in Cochrane and Campbell reviews of psychosocial treatment efficacy with substance use disorders

Abstract

Background and AimsA crucial, but under-appreciated, aspect in experimental research on psychosocial treatments of substance use disorders concerns what kinds of control groups are used. This paper examines how the distinction between different control-group designs have been handled by the Cochrane and the Campbell Collaborations in their systematic reviews of psychosocial treatments of substance abuse disorders.MethodsWe assessed Cochrane and Campbell reviews (n = 8) that were devoted to psychosocial treatments of substance use disorders. We noted what control groups were considered and analysed the extent to which the reviews provided a rationale for chosen comparison conditions. We also analysed whether type of control group in the primary studies influenced how the reviews framed the effects discussed and whether this was related to conclusions drawn.ResultsThe reviews covered studies involving widely different control conditions. Overall, little attention was paid to the use of different control groups (e.g. head-to-head comparisons versus untreated controls) and what this implies when interpreting effect sizes. Seven of eight reviews did not provide a rationale for the choice of comparison conditions.ConclusionsCochrane and Campbell reviews of the efficacy of psychosocial interventions with substance use disorders seem to underappreciate that the use of different control-group types yields different effect estimates. Most reviews have not distinguished between different control-group designs and therefore have provided a confused picture regarding absolute and relative treatment efficacy. A systematic approach to treating different control-group designs in research reviews is necessary for meaningful estimates of treatment efficacy.

Systematic review of control groups in nutrition education intervention research

“…A systematic review of psychosocial interventions with substance use disorders also found studies gave little justification for control group choice or considerations for how this choice may have affected study outcomes [24]. …”

“…(−) Comparing control and experimental condition is reasonable only if both are blind to treatment group assignment [2, 92]. (−) Additional resource input for control condition development; using alternative active treatment when the effect of attention on participant outcome is unknown may be an unnecessary expense.(−) Rigorous control of non-specific treatment tends to contribute to study effects (i.e., control participant improvement), thus larger sample sizes or an increased risk of Type 1 error (e.g., p-level set higher than typical <0.05) is needed to prevent erroneously rejecting effective interventions as ineffective and to detect potentially small yet clinically important effect sizes [1, 9, 13, 17, 24]. Dismantling (or Additive) Component Attention Control: Typically used with a multi-part intervention where the individual parts are separated to identify which are most salient to the outcomes (often with the goal of increasing cost-effectiveness by paring down intervention parts).…”

Effects of psychosocial interventions on behavioural problems in youth: A close look at Cochrane and Campbell reviews

“…In fact, bona fide treatments for youth problems appear to be only marginally more effective than psychotherapy‐based treatment‐as‐usual (TAU) when controlling for confounders such as dose and researcher allegiance (Spielmans, Gatlin, & McFall, ). However, the tendency for researchers to juxtapose active and passive control groups in research reviews (Karlsson & Bergmark, ) means that we may still lack basic knowledge about absolute and relative treatment efficacy.…”

“…As part of a broader project addressing the production of evidence for psychosocial interventions in different areas (Bergmark, Skogens, & von Greiff, ; Karlsson & Bergmark, ; Karlsson, Bergmark, & Lundström, ), in this article we take a closer look at the available evidence for the effects of this set of interventions, focusing on systematic research reviews. Besides focusing on effects, we address the potential of selective reporting of positive findings in the reviews, the degree of research allegiance, control groups considered in the reviews and what this may imply regarding ‘specificity’ (Magill & Longabaugh, ) of the effect estimates.…”

“…Prior studies from different areas have concluded that Cochrane reviews are of superior methodological quality to reviews published in scientific journals (Delaney et al, ; Jadad et al, ; Moseley, Elkins, Herbert, Maher, & Sherrington, ). However, a previous analysis of Cochrane and Campbell reviews on psychosocial treatments for substance use disorders illustrates several flaws (Karlsson & Bergmark, ), so there may be reason to assume similar problems in reviews on behavioural problems in youth as well. It should be stressed that we do not intend to critique systematic reviews as such – on the contrary, we believe that they are the best tools available for systematising research evidence – but rather to focus on crucial methodological issues in these.…”