Human fetal development has been associated with brain health at later stages. It is unknown whether and how consistently growth in utero, as indexed by birth weight (BW), relates to lifespan brain characteristics and changes, and to what extent these influences are of a genetic and/or environmental nature. We hypothesized that associations of BW and structural brain characteristics persist through the lifespan, with topographically consistent effects across samples of varying age and origin, that BW is not protective against atrophy in aging, and that effects are partly environmental. The associations of BW and cortical area, thickness, volume and their change were investigated vertex-wise in developmental (ABCD), older adult and aging (UKB) and lifespan (LCBC) longitudinal samples. In total, 5794 persons (4-82 years, w/ 386 monozygotic twins), were followed for up to 8.3 years, yielding 12,088 brain MRIs. Positive associations between BW and cortical surface area and volume were remarkably stable through the lifespan, within and across samples of different origin, with spatial correlations in the range r = .51- .79. In contrast, there was modest and no consistent effect of BW on brain changes. Effects of BW discordance in the monozygotic twin subsample indicated the effects were partly environmental. In conclusion, the influence of prenatal growth on cortical topography is stable through the lifespan, and is reliably seen in development, adulthood, and aging. These findings support early life influence on the brain through the lifespan according to a threshold model of brain reserve, rather than a maintenance model.