Brain age is becoming a widely applied imaging-based biomarker of neural aging and potential proxy for brain integrity and health. We estimated multimodal and modality-specific brain age in the Whitehall II MRI cohort using machine learning and imaging-derived measures of gray matter morphology, diffusion-based white matter microstructure, and resting state functional connectivity. Ten-fold cross validation yielded multimodal and modality-specific brain age estimates for each participant, and additional predictions based on a separate training sample was included for comparison. The results showed equivalent age prediction accuracy between the multimodal model and the gray and white matter models (R 2 of 0.34, 0.31, and 0.31, respectively), while the functional connectivity model showed a lower prediction accuracy (R 2 of 0.01). Cardiovascular risk factors, including high blood pressure, alcohol intake, and stroke risk score, were each associated with more apparent brain aging, with consistent associations across modalities.