Age-related memory impairments have been associated with structural changes in the dopaminergic system, but the underlying mechanisms remain unclear. Recent work indicates that iron accumulation might be of particular relevance. As iron accumulates, a degeneration of myelin sheaths has been observed in the elderly, but the relationship between both and their impact on memory performance in healthy elderly humans remain important open questions. To address this issue, we combined an established behavioral paradigm to test memory performance [verbal learning memory test (VLMT)] with state of the art quantitative magnetic resonance imaging techniques allowing us to quantify the degree of myelination and iron accumulation via markers of tissue microstructure in a group of young (18 -32 years) and healthy elderly humans (55-79 years). As expected, we observed a decrease in gray matter volume and myelin, and an increase of iron in the elderly relative to the young subjects within widespread brain regions, including the basal ganglia. Furthermore, higher levels of iron within the ventral striatum were accompanied by a negative correlation between myelin and iron specific for the elderly participants. Importantly, both markers of iron and myelin (and their ratio) predicted the performance of the elderly in the VLMT. This suggests that ventral striatum iron accumulation is linked to demyelination and impairments in declarative memory. Together, our data provide novel insights into underlying microstructural mechanisms of memory decline in the elderly.