In contrast to postmenopausal women, evidence for a favorable effect of exercise on Bone Mineral Density (BMD) is still limited for men. This might be due to the paucity of studies, but also to the great variety of participants and study characteristics that may dilute study results. The aim of the present systematic review and meta-analysis was to evaluate the effect of exercise on BMD changes with rational eligibility criteria. A comprehensive search of six electronic databases up to 15 March 2021 was conducted. Briefly, controlled trials ≥6 months that determined changes in areal BMD in men >18 years old, with no apparent diseases or pharmacological therapy that relevantly affect bone metabolism, were included. BMD changes (standardized mean differences: SMD) of the lumbar spine (LS) and femoral neck (FN) were considered as outcomes. Twelve studies with 16 exercise and 12 control groups were identified. The pooled estimate of random-effect analysis was SMD = 0.38, 95%-CI: 0.14–0.61 and SMD = 0.25, 95%-CI: 0.00–0.49, for LS and FN, respectively. Heterogeneity between the trials was low–moderate. Funnel plots and rank and regression correlation tests indicate evidence for small study publication bias for LS but not FN-BMD. Subgroup analyses that focus on study length, type of exercise and methodologic quality revealed no significant difference between each of the three categories. In summary, we provided further evidence for a low but significant effect of exercise on BMD in men. However, we are currently unable to give even rough exercise recommendations for male cohorts.