Differences in causal estimates from longitudinal analyses of residualized versus simple gain scores: Contrasting controls for selection and regression artifacts

Abstract

This study estimates the causal effects of six corrective actions for children's problem behaviors, comparing four types of longitudinal analyses that correct for pre-existing differences in a cohort of 1,464 4- and 5-year-olds from Canadian National Longitudinal Survey of Children and Youth (NLSCY) data. Analyses of residualized gain scores found apparently detrimental effects of all corrective actions by parents and professionals on subsequent antisocial behavior and hyperactivity. In contrast, analyses of simple gain scores found only apparently beneficial effects. Temporally reversed analyses yielded the same pattern of results, consistent with selection biases and regression artifacts, not with unidirectional causal effects. The findings were similar for corrective actions by professionals (e.g., Ritalin, psychotherapy) and by parents (physical and nonphysical punishment, scolding/yelling, “hostile-ineffective” parenting). Longitudinal analyses should check for similar artifacts by implementing temporally-reversed analyses and by determining whether causally relevant coefficients would replicate without artifacts biased in their favor.

Do nonphysical punishments reduce antisocial behavior more than spanking? a comparison using the strongest previous causal evidence against spanking

“…Moreover, the non-significant associations generally reversed signs when predicting simple gain scores in the latent variable of externalizing behavior problems, consistent with small residual selection biases[44,48,56]. These results suggest that the findings of all three types of analyses are due to residual selection biases that are minimized by controlling for a latent variable for externalizing behavior problems to reduce measurement error and to maximize the comprehensiveness of the proxy for the selection process.…”

Making Valid Causal Inferences About Corrective Actions by Parents From Longitudinal Data

“…No disciplinary tactic was ever associated with reduced child behavior problems, and 7 of the 12 tactics predicted significantly worse behavior problems in at least one analysis. Moreover, two of those studies found that two corrective actions by professionals, psychotherapy and Ritalin, both looked as harmful as spanking when analyzed in the same way (see Table 1; Larzelere, Cox, & Smith, 2010;Larzelere, Ferrer, Kuhn, & Danelia, 2010). Gershoff et al (2010).…”

Many Replications Do Not Causal Inferences Make

“…This finding has been replicated at least eight times, yielding a small but significant mean of β = 0.07 with antisocial types of behavior problems in a recent meta-analysis (Ferguson, 2013). Two of those replications found similar adverse effects for nonphysical punishments, psychotherapy, and Ritalin on antisocial behavior or hyperactivity when controlling for initial differences on those outcomes (Larzelere, Cox, & Smith, 2010;Larzelere, Ferrer, et al, 2010). In the first major evaluation of Head Start, Cicirelli (1969) also found adverse effects on subsequent academic success even after using a matched comparison group and controlling statistically for socioeconomic status.…”

“…Second, this pattern of contradictory effects for ANCOVA-type analyses in comparison with analyses of simple change scores has been shown for other corrective actions, including out-ofhome placements (Berger, Bruch, Johnson, James, & Rubin, 2009) and job-training programs (LaLonde, 1986). Third, one study showed that this same pattern of contradictory results replicates after reversing the waves of data (Larzelere, Ferrer, et al, 2010)-an old strategy for discriminating between true causal effects and statistical artifacts (Galton, 1886). These examples illustrate the importance of checking the robustness of findings across alternative statistical analyses, which is a practice much more typical of economics than of developmental psychology (Duncan, Engel, Claessens, & Dowsett, 2014).…”

“…First, in contrast to analysis of covariance (ANCOVA) types of controls, predictions of simple change scores have shown that all five types of corrective actions are associated with improvements in subsequent outcomes, although those improvements were significant only for nonphysical punishment (Larzelere, Ferrer, et al, 2010) and Head Start (Lee, Brooks-Gunn, & Schnur, 1988). These findings occur partly because predictions of simple gain scores are biased in favor of corrective actions due to regression toward the mean, whereas ANCOVAtype controls are biased against them because covariates measure biases imperfectly (Angrist & Pischke, 2009;Larzelere, Ferrer, et al, 2010). Second, this pattern of contradictory effects for ANCOVA-type analyses in comparison with analyses of simple change scores has been shown for other corrective actions, including out-ofhome placements (Berger, Bruch, Johnson, James, & Rubin, 2009) and job-training programs (LaLonde, 1986).…”