Increases in harmful drinking among older adults indicate the need for a more thorough understanding of the relationship between later‐life alcohol use and brain health. The current study investigated the relationships between alcohol use and progressive grey and white matter changes in older adults using longitudinal data. A total of 530 participants (aged 70 to 90 years; 46.0% male) were included. Brain outcomes assessed over 6 years included total grey and white matter volume, as well as volume of the hippocampus, thalamus, amygdala, corpus callosum, orbitofrontal cortex and insula. White matter integrity was also investigated. Average alcohol use across the study period was the main exposure of interest. Past‐year binge drinking and reduction in drinking from pre‐baseline were additional exposures of interest. Within the context of low‐level average drinking (averaging 11.7 g per day), higher average amount of alcohol consumed was associated with less atrophy in the left (B = 7.50, pFDR = 0.010) and right (B = 5.98, pFDR = 0.004) thalamus. Past‐year binge‐drinking was associated with poorer white matter integrity (B = −0.013, pFDR = 0.024). Consuming alcohol more heavily in the past was associated with greater atrophy in anterior (B = −12.73, pFDR = 0.048) and posterior (B = −17.88, pFDR = 0.004) callosal volumes over time. Across alcohol exposures and neuroimaging markers, no other relationships were statistically significant. Within the context of low‐level drinking, very few relationships between alcohol use and brain macrostructure were identified. Meanwhile, heavier drinking was negatively associated with white matter integrity.