Impact of experimental design on effect size: Findings from the research literature on training.

Abstract

This article examines whether differences in the equations commonly used to calculate effect size for single group pretest-posttest (SGPP) designs versus those for control group designs can account for the finding that SGPP designs yield larger mean effect sizes (e.g., M. S. Lipsey & D. B. Wilson, 1993). It was found that the assumptions of no control group effect and the equivalence of pretraining and posttraining dependent variable standard deviations required for these equations to produce equivalent estimates of effect size were violated for some dependent variable types. Results indicate that control group effects and inflation in the standard deviation of the posttraining dependent variable measure account for most of the observed difference in effect size. The most severe violations occurred when the dependent variable was a knowledge assessment. Methods for including data from SGPP designs in meta-analyses that minimize potential biases are discussed.In an extensive review of psychological, educational, and behavioral treatment research, Lipsey and Wilson (1993) examined results from meta-analyses reporting effect size estimates for studies subgrouped by research design. Data from studies employing single group pretest-posttest (SGPP) designs were compared with data from studies using control or comparison group designs (i.e., pretestposttest with control, PPWC; and posttest only with control, POWC). Across 45 meta-analyses, mean effect sizes (dvalues) for SGPP designs were 0.29 SDs larger (0.76 versus 0.47) than effect sizes for studies using control or comparison group designs. Research by Guzzo, Jette, and Katzell (1985) has found similar differences. The common conclusion among these researchers is that data from SGPP designs "clearly are capable of upwardly biasing the mean treatment effect estimates derived from meta-analysis" (Lipsey & Wilson, 1993, p. 1194.Lipsey and Wilson's (1993) solution to the perceived upward bias in effect size for SGPP designs was not to

The influence of experimental design on the magnitude of the effect size -peer tutoring for elementary, middle and high school settings: A meta-analysis

“…To date, only Carlson and Schmidt (1999) have compared the reported ESs between two homogeneous groups, single pre-post treatment-step design versus experimental control designs within a single area of training. Their conclusion was that experimental control designs studies yielded lower reported ESs.…”

A Test of Major Assumptions About Behavior Change: A Comprehensive Look at the Effects of Passive and Active HIV-Prevention Interventions Since the Beginning of the Epidemic.

“…This measure controls for the inflation in the standard deviation following treatment (for an excellent analysis of the problem, see Carlson & Schmidt, 1999). Effect sizes were also derived from exact reports of t tests, F ratios, proportions, p values, and confidence intervals.…”

“…Becker's (1988) g, which is calculated by subtracting the mean at the posttest from the mean at the pretest and dividing the difference by the standard deviation of the pretest measure. This measure controls for the inflation in the standard deviation following treatment (for an excellent analysis of the problem, see Carlson & Schmidt, 1999). Effect sizes were also derived from exact reports of t tests, F ratios, proportions, p values, and confidence intervals.…”

The Incredible Years Teacher Classroom Management Programme in Kindergartens: Effects of a Universal Preventive Effort

“…For the whole sample, we adopted an approach from Hedges (2007) to take the clustering within kindergarten teachers into account, using Hedges' dT2 measure. In both cases, we used the pooled standard deviation for the groups at Time 1 in the standardization (Morris 2008;Carlson & Schmidt 1999). We used SPSS version 23 for all analyses.…”