Risky business: Correlation and causation in longitudinal studies of skill development.

Abstract

Developmental theories often posit that changes in children's early psychological characteristics will affect much later psychological, social, and economic outcomes. However, tests of these theories frequently yield results that are consistent with plausible alternative theories that posit a much smaller causal role for earlier levels of these psychological characteristics. Our paper explores this issue with empirical tests of skill building theories, which predict that early boosts to simpler skills (e.g., numeracy or literacy) or behaviors (e.g, anti-social behavior or executive functions) support the long-term development of more sophisticated skills or behaviors. Substantial longitudinal associations between academic or socioemotional skills measured early and then later in childhood or adolescence are often taken as support of these skill-building processes. Using the example of skill-building in mathematics, we argue that longitudinal correlations, even if adjusted for an extensive set of baseline covariates, constitute an insufficiently risky test of skill-building theories. We first show that experimental manipulation of early math skills generates much smaller effects on later math achievement than the nonexperimental literature has suggested. We then conduct falsification tests that show puzzlingly high cross-domain associations between early math and later literacy achievement. Finally, we show that a skill-building model positing a combination of unmeasured stable factors and skillbuilding processes is able to reproduce the pattern of experimental impacts on children's mathematics achievement. Implications for developmental theories, methods, and practice are discussed.Keywords early childhood; interventions; skill-building; cognitive development; education Developmental theories often posit that changes in children's early psychological characteristics will affect their much later psychological, social, and economic outcomes. Such theories include skill-building theories (e.g., Baroody, 1987;Stanovich, 1986;Cunha & Heckman, 2007), theories of the life-course development of psychopathology (e.g., Moffitt, 1993), theories that posit reciprocal effects between children and their environments Corresponding Author: Drew H. Bailey, School of Education, 3200 Education, University of California, Irvine, Irvine, CA 92697-5500, dhbailey@uci.edu. HHS Public AccessAuthor manuscript Am Psychol. Author manuscript. Author Manuscript Author ManuscriptAuthor Manuscript Author Manuscript (e.g., Scarr & McCartney, 1983), and theories of early critical periods in children's social and cognitive development (Fraley and Roisman, 2015).Tests of these theories are often conducted by estimating correlations between important outcomes and children's early psychological characteristics that have been adjusted by statistical controls for variables that might affect both early and later child characteristics. These findings are given varying degrees of causal interpretation. A cautious, yet superficial, alternative app...

Prevention: Necessary But Insufficient? A 2‐Year Follow‐Up of an Effective First‐Grade Mathematics Intervention

“…We found that projected impacts systematically over‐estimated observed impacts, replicating Bailey et al. (). Thus, given the implausibility of regularly collecting a battery of pretest controls more thorough than those included in this study, regression control does not appear to provide an effective means for projecting the future benefits of successful interventions.…”

“…The extent to which these findings generalize to other types of interventions, target skills, and age groups is not well understood. The finding that intervention effects declined to approximately 1/3 their initial size after the first year is similar to findings from prior comparisons of fadeout effects across multiple studies of academic interventions in early childhood and the early school years (Bailey et al., ; Li et al., ). It is less clear if this pattern generalizes well to interventions targeting older children.…”

“…The present analysis extends prior work with a rigorous set of pretest control measures, in a sample with which observed impacts can be compared with projections based on end-of-intervention impacts and regression-estimated persistence rates. We found that projected impacts systematically over-estimated observed impacts, replicating Bailey et al (2018). Thus, given the implausibility of regularly collecting a battery of pretest controls more thorough than those included in this study, regression control does not appear to provide an effective means for projecting the future benefits of successful interventions.…”

“…When they are followed, fadeout is common. The typical finding is rapid declines in treatment effects soon after the program has ended and small to no long-term advantages relative to at-risk children who did not receive the intervention (Bailey, Duncan, Watts, Clements, & Sarama, 2018;Li et al, 2017).…”

The relation between subitizable symbolic and non‐symbolic number processing over the kindergarten school year

“…different measures were administered at different time points), the authors do not implement repeated measures and are therefore unable to control for the preexisting relation between symbolic and non‐symbolic processing at time one. As such, these studies do not provide strong evidence for a relation between non‐symbolic skills at time one and growth in symbolic number abilities (see Bailey, Duncan, Watts, Clements, & Sarama, ).…”