A Monte Carlo Comparison Study of the Power of the Analysis of Covariance, Simple Difference, and Residual Change Scores in Testing Two-Wave Data

Abstract

This study compares the analysis of covariance (ANCOVA), difference score, and residual change score methods in testing the group effect for pretest–posttest data in terms of statistical power and Type I error rates using a Monte Carlo simulation. Previous research has mathematically shown the effect of stability of individual scores from pretest to posttest, reliability, and nonrandomization (i.e., pretest imbalance) on the performance of the ANCOVA, difference score, and residual change score methods. However, related power issues have not been adequately addressed. The authors examined the impact of stability of measurement over time, reliability of covariate and criterion, nonrandomization, sample size, and treatment effect size on statistical power of the three methods. Across conditions, ANCOVA and residual change score methods had similar power rates. When reliability was less than perfect, ANCOVA had more power than the difference score method when there was an increase from pretest to posttest and a positive baseline imbalance (i.e., treatment group had higher pretest scores than the control group), or when there was a decrease from pretest to posttest and a negative baseline imbalance, and vice versa. In case of perfect reliability, the statistical power of ANCOVA did not differ from the difference score method. For the difference score method, when reliability was low, there was no effect of stability on power, whereas when reliability was high or perfect, power increased as stability increased for medium and large effect sizes. Difference scores may be preferred over ANCOVA under certain circumstances.

The influence of future time perspective on work engagement and job performance: the role of job crafting

“…A positive residual score indicates an increase in FTP, job crafting, work engagement, or job performance over the 1-year study period that could not be predicted from the baseline scores. Residual change scores are typical for two-occasion longitudinal studies (KisbuSakarya, MacKinnon, & Aiken, 2013;Lipshits-Braziler, Gati, & Tatar, 2015) and are preferred when one is interested in statistically partialling the influence of the score on the first assessment from the score on the second assessment (Salthouse & Tucker-Drob, 2008). The residual change score is widely considered superior to raw change scores (i.e., T2 minus T1).…”

Comparing Models of Change to Estimate the Mediated Effect in the Pretest–Posttest Control Group Design

“…In the case of non-randomized or observational studies, adjusting for pretest scores require more careful consideration in order to ensure accurate estimation of treatment effects (Morgan & Winship, 2014). It is known from previous research that ANCOVA and difference score models can lead to very different results regarding change across two-waves of data and represent different theoretical models of change (Jamieson, 1999; Kisbu-Sakarya, et al, 2013; Lord, 1967; Wright, 2006) but further work is needed in order to examine which of the four longitudinal models discussed in this article would perform the best for estimating the mediated effect when systematic preexisting differences exist.…”

“…Researchers have explored the results of violating these assumptions using ANCOVA and difference score models outside the context of mediation (Jamieson, 1999; Kisbu-Sakarya, MacKinnon, & Aiken, 2013; Van Breukelen, 2006, 2013; Wright, 2006). For the application of mediation, VanderWeele and Vansteelandt (2009) describe the four assumptions necessary for identification of a mediated effect:

No unmeasured confounders of the relation between X and Y .

No unmeasured confounders of the relation between M and Y .

No unmeasured confounders of the relation between X and M .

No measured or unmeasured confounders of M and Y that have been affected by treatment.

…”

“…No treatment group variable is included in the regression of posttest scores on pretest scores which means posttest scores for units in both treatment groups are adjusted for pretest scores based on an aggregate of pretest scores across both treatment groups (e.g., Cronbach & Furby, 1970; Kisbu-Sakarya et al, 2013) which may lead to erroneous inference regarding group differences in some cases (Maxwell, Delaney, & Manheimer, 1985). …”

The power of a paired t-test with a covariate

“…This paired t -test is a one sample (population) t -test on the mean difference the stems from Gossett’s work on small sample tests of means (Student, 1908) and is a classic method to test gains from pretests to posttests (Lord, 1956; McNemar, 1958). The analysis of covariance (ANCOVA) literature also has long recognized the gains in statistical power to measure differences when using covariates (Cochran, 1957; Kisbu-Sakarya, MacKinnon, & Aiken, 2013; Oakes & Feldman, 2001; Porter & Raudenbush, 1987). Previous work has examined the use of covariates to estimate differential gains based on the initial value (Garside, 1956), but to our knowledge a factor for predicting the gains in precision for the mean difference has yet to be developed.…”