In this diffusion tension imaging study, voxel wise structural equation modeling was used to unravel the relation between white matter, cognition, and age. Four neurocognitive ageing models describing the interplay between age, white matter integrity, and cognition were investigated but only two models survived an Akaike information criterion-based model selection procedure. The independent factor model predicts that there is no relation between white matter integrity and cognition although both systems are affected by age. The cognitive mediation model predicts that the relation between age and white matter integrity is mediated through cognition. Roughly 60% of the observed voxels were in agreement with the independent factor model while 16% of the observed voxels were in agreement with the cognitive mediation model. Imaging results of the latter model suggest that the deterioration of fibers—that connect the two hemispheres with each other—is partly caused by an age-related decline in cognitive functioning.