Previous magnetic resonance imaging (MRI) studies described consistent age-related gray matter (GM) reductions in the fronto-parietal neocortex, insula and cerebellum in elderly subjects, but not as frequently in limbic/paralimbic structures. However, it is unclear whether such features are already present during earlier stages of adulthood, and if age-related GM changes may follow non-linear patterns at such age range. This voxel-based morphometry study investigated the relationship between GM volumes and age specifically during non-elderly life (18–50 years) in 89 healthy individuals (48 males and 41 females). Voxelwise analyses showed significant (p < 0.05, corrected) negative correlations in the right prefrontal cortex and left cerebellum, and positive correlations (indicating lack of GM loss) in the medial temporal region, cingulate gyrus, insula and temporal neocortex. Analyses using ROI masks showed that age-related dorsolateral prefrontal volume decrements followed non-linear patterns, and were less prominent in females compared to males at this age range. These findings further support for the notion of a heterogeneous and asynchronous pattern of age-related brain morphometric changes, with region-specific non-linear features.