Random versus nonrandom assignment in controlled experiments: Do you get the same answer?

Abstract

Psychotherapy meta-analyses commonly combine results from controlled experiments that use random and nonrandom assignment without examining whether the 2 methods give the same answer. Results from this article call this practice into question. With the use of outcome studies of marital and family therapy, 64 experiments using random assignment yielded consistently higher mean posttest effects and less variable posttest effects than 36 studies using nonrandom assignment. This difference was reduced by about half by taking into account various covariates, especially pretest effect size levels and various characteristics of control groups. The importance of this finding depends on (a) whether one is discussing meta-analysis or primary experiments, (b) how precise an answer is desired, and (c) whether some adjustment to the data from studies using nonrandom assignment is possible. It is concluded that studies using nonrandom assignment may produce acceptable approximations to results from randomized experiments under some circumstances but that reliance on results from randomized experiments as the gold standard is still well founded.

Randomised controlled trials for policy interventions: a review of reviews and meta-regression

“…The factors falling outside of these categories were publication status 46,72,74 and year of publication. 74,75 Some studies gave a clear rationale for the variables they selected as potential effect modifiers, 72,74 and other studies gave detailed descriptions of each variable of interest.…”

“…They included interventions for preventing juvenile delinquency, 42,46 treatment of alcohol abuse 73,87 and psychological interventions. [74][75][76] There was an overlap with the health field, with Heinsman and Shadish 72 including interventions in the mental health and health-care fields.…”

“…• according to study design 42 • according to study design but also taking into account other variables that might be confounded with design 46,72,74,87 • according to methods (including design) and substantive factors (related to the intervention) 76 • according to some other variable of primary interest, but also assessing study design. 75 The review of reviews by Glazerman et al 70,71 did not aim to investigate the influence of effect modifiers, but did investigate differences between randomised and non-randomised methods, and the role of different techniques for comparing randomised and non-randomised groups within a single study.…”

“…After sifting we identified one review of withinstudy comparisons of randomised and nonrandomised participants, 70,71 six single metaanalyses 42,46,[72][73][74][75] and one review of meta-analyses. 76 These eight studies form the basis of this chapter.…”

The Metric Comparability of Meta-Analytic Effect-Size Estimators From Factorial Designs.

“…A problem encountered by every meta-analyst is how to obtain estimates of the size of a psychological effect that are comparable with each other, from studies in the literature that have used different statistical designs (Cooper & Hedges, 1994;Glass, McGaw, & Smith, 1981;Hedges & Olkin, 1985;Holmes, 1984;Hunter & Schmidt, 1990;Rosenthal, 1991Rosenthal, , 1994Shadish & Ragsdale, 1996;Wolf, 1986).…”