During the course of adulthood and ageing, white matter (WM) structure and organisation are characterised by slow degradation processes such as demyelination and shrinkage. An acceleration of such ageing process has been linked to the development of a range of diseases. Thus, an accurate description of healthy brain maturation, in particular, in terms of WM features, provides a cornerstone in the understanding of ageing. We use longitudinal diffusion magnetic resonance imaging to provide an overview of WM changes at different spatial and temporal scales in the UK Biobank (UKB) (N=2,678; age[scan 1]=62.38(7.23) years; age[scan 2]=64.81(7.1) years). To examine the genetic overlap between WM structure and common clinical conditions, we tested the associations between WM structure and polygenic risk scores (PGRS) for the most common neurodegenerative disorder, Alzheimer's disease, and common psychiatric disorders (uni- and bipolar depression, anxiety, obsessive-compulsive, autism, schizophrenia, attention-deficit-hyperactivity) in longitudinal (N=2,329) and cross-sectional UKB validation data (N=31,056). Global and regional single and multi-compartment fractional anisotropy, intra-axonal water fraction, and kurtosis metrics decreased (mean beta=-0.04),whereas diffusivity metrics, and free water increased with age (mean beta=0.05), with the annual rate of WM change (ARoC) accelerating at higher ages for both global (mean beta=0.01) and regional WM metrics (mean beta=0.01). Voxel-level trends indicated decreasing anisotropy, and variable spatial patterns for other diffusion metrics, suggesting differential changes in frontal compared to other brain regions. Although effect sizes were small (mean absolute beta[all]=0.01), ARoC in middle cerebral peduncle WM had the strongest association with PGRS, especially for Alzheimer's: mean absolute beta=0.01. PGRS were more strongly related to ARoC than cross-sectional measures (d[scan 1]=0.03, d[scan 2]=0.03, d[validation]=0.03). Our findings indicate spatially distributed WM changes across the brain, as well as distributed associations of PGRS with WM. Importantly, brain longitudinal changes reflected the genetic risk for disorder development better than the utilised cross-sectional measures, with regional differences giving more specific insights into gene-brain change associations than global averages.