Using Dissertations to Examine Potential Bias in Child and Adolescent Clinical Trials.

Abstract

The major youth psychotherapy meta-analyses have relied on published studies, which may have led to biased effect size estimates. To examine this possibility, the authors compared 121 dissertations with 134 published studies and found the following: (a) few differences on individual methodological variables, but, overall, stronger methodology in dissertations; (b) no differences in the steps taken to ensure treatment integrity; and (c) a mean dissertation effect size less than half that of published studies. The effect size difference remained robust across tests controlling for all reliable method differences. The findings suggest that dissertations are so strong, both methodologically and clinically, that they warrant inclusion in child psychotherapy meta-analyses and that previous meta-analyses, by excluding them, may have overestimated treatment effects.

Effects of psychotherapy for depression in children and adolescents: A meta-analysis.

“…To reduce the possibility of a publication bias favoring positive results, which threatens the validity of meta-analyses (Cook et al, 1993;McLeod & Weisz, 2004;Sohn, 1996), we included non-peer-reviewed studies (e.g., book chapters) and doctoral dissertations. When separate articles were published from the same data set (e.g., articles presenting posttreatment and follow-up findings separately, or dissertations that were later published in journals), these were combined for analysis as a single study.…”

“…In the youth psychotherapy literature generally, unpublished/non-peerreviewed studies show significantly lower ES than published studies (see McLeod & Weisz, 2004). However, most youth depression psychotherapy research is relatively recent compared with treatment research with other youth conditions (see Weisz, Hawley, & Jensen Doss, 2004); more recent research may profit from an increased focus, in journal reviews, on the …”

“…We have now identified more than twice as many randomized trials as the most complete meta-analysis to date (Michael & Crowley, 2002). Second, it should be noted that two of the prior meta-analyses Reinecke et al, 1998aReinecke et al, ,1998b included only articles published in peer-reviewed journals.Including only such peer-reviewed work may increase the risk of ES overestimation because the peer review process favors positive findings (see, e.g., McLeod & Weisz, 2004). Third, for those studies that the meta-analysts did identify, ES computation appears to have relied solely on data in the published reports, on estimates of effects for the numerous studies not reporting sufficient data, and on exclusion of studies (e.g., 50% of the studies identified, in one of the meta-analyses), rather than on contacting authors and persuading them to provide the information needed for precise ES values.…”

Guest Editor’s Introduction to Special Issue: The Science and Practice of Research Synthesis

“… publication status is not a good proxy for study quality (McLeod & Weisz, 2004); and  the odds of publication are 2 to 3 times greater for studies that have positive, statistically significant results (Song, Parekh, et al, 2010;Song, Parekh-Bhurke, et al, 2009).…”

Challenges in Evaluating Psychosocial Interventions for Autistic Spectrum Disorders

“…In much of medicine, the view of a single RCT as providing a completely decisive answer is now being replaced by one in which a study is seen as contributing to a larger body of evidence, by meeting standards that allow inclusion in a systematic review (McLeod & Weisz, 2004). All treatment studies should be designed in such a way that they can contribute to a greater program of knowledge accumulation beyond whether a particular treatment is more effective than another treatment in a particular sample.…”