What Is the Evidence for Evidence-Based Treatments? A Hard Look at Our Soft Underbelly

Abstract

In the rising quest for evidence-based interventions, recent research often does not give adequate attention to "nonspecific therapeutic factors," including the effects of attention, positive regard, and therapeutic alliance, as well as the effects of treatment dose, intensity and actual processes mediating therapeutic change. To determine the extent to which recent clinical trial designs fully this problem, the authors conducted a systematic review of Psych-Lit/Medline of all controlled child psychotherapy treatment studies from 1995 to 2004. A total of 52 studies were identified that met review criteria: two or more therapy conditions and random assignment of participants to intervention groups. Of the 52 studies, one group (n = 29) compared a presumably active treatment with 1 or more similarly intensive treatments (often an attention control group) presumably not containing the active therapeutic ingredients. Of these, 14 studies found evidence of consistent differences between the two groups, whereas 15 did not. An additional group of studies (n = 27) compared therapy groups with different levels of intensity and "dose" of the putatively active treatment; 13 of these found evidence of the effects of different levels of treatment dose/intensity on outcomes and 14 did not. Four studies met criteria for inclusion in both groups. Across both groups of studies, when positive effects were found, few studies systematically explored whether the presumed active therapeutic ingredients actually accounted for the degree of change, nor did they often address plausible alternative explanations, such as nonspecific therapeutic factors of positive expectancies, therapeutic alliance, or attention. Findings suggest that many child psychotherapy treatment studies have not inadequately controlled for nonspecific factors such as attention and treatment intensity and have failed to assess specific mediators of change. Specific recommendations for future studies are offered, specifically: 1. Initial specification in study design how investigators will test if the intervention is efficacious over and above the effects of positive expectancies, positive regard, or attention; 2. Planned (rather than post hoc) analyses to explore whether and how a given treatment's specific ingredients or the overall intensity of its "dose" are related to treatment outcomes; 3. More studies conducting head-to-head tests of different types but equally credible forms of treatment, with planned analyses testing different mechanisms of change 4. When positive effects of a treatment vs. a control are found, systematic elimination of specific, competing hypotheses of reasons for treatment efficacy (e.g., attention, therapeutic alliance, face validity of treatment and client's treatment expectations, changes in self-efficacy), and 5. When negative effects are reported, appropriate analyses to address alternative explanations (lack of power, floor or ceiling effects, mediator analyses to address possible fidelity or adherence problems, therapist effects, abs...

Effects of psychotherapy for depression in children and adolescents: A meta-analysis.

“…A concern raised by some (e.g., Jensen, 2003;Weisz, 2004) about the evidence on youth treatment research in general is that so many of the trials have compared active treatment with passive control conditions, including no treatment and waitlist. This was evident in the current depression study set as well, with 20 of the 35 studies having used passive control conditions.…”

“…A potential weakness of most psychotherapy outcome research, including youth depression research, is that it rests on comparisons of active treatments with inert conditions such as waitlist or no treatment (see Jensen, 2003;Weisz, 2004). Such inert conditions control only for the passage of time and the natural time course of problems and disorders, not for placebo or expectancy effects and not for such nonspecific effects as improvement due to attention or to the benefits of a therapeutic relationship.…”

“…What is not clear from the available data is whether this picture results from a low-potency version of CBT in TADS, from the unusual and challenging comparison of CBT with a medication placebo condition in TADS (see Baskin et al, 2003), from a combination of the two, or from other factors not identified. A concern raised by some (e.g., Jensen, 2003;Weisz, 2004) about the evidence on youth treatment research in general is that so many of the trials have compared active treatment with passive control conditions, including no treatment and waitlist. This was evident in the current depression study set as well, with 20 of the 35 studies having used passive control conditions.…”

Effects of a School-based Intervention for Urban Adolescents with Asthma

“…The control group was a wait-list control group who received no attention. Although unlikely, it is possible that treatment benefits of ASMA were due to ''nonspecific therapeutic factors,'' including the effects of attention and positive regard received by those in the intervention condition (52). At the same time, treatment differences may have been mitigated (1) if the consent process and/or surveys fostered attention to treating asthma in the control students; (2) if participants received interim treatments from other sources, which we did not assess; and/or (3) if there was contamination of control subjects by the treated adolescents.…”

Moderators and Mediators of Treatment Outcome for Youth With ADHD: Understanding for Whom and How Interventions Work

“…Indeed, interventions for children and adolescents that appear to effect change because of certain program ingredients may actually produce benefits for very different reasons. 2 In short, outcome research will be enriched by investigation of relevant subgroups with particularly good (or poor) treatment response and by examination of relevant processes and mechanisms that yield clinically significant change. Without knowledge of the particular subgroups that respond best and worst to any particular treatment, and without systematic understanding of treatment mechanisms, research on effective treatments is bound to remain at a relatively primitive, descriptive level.…”