Controversial results have been reported about the association of calcium, magnesium, and phosphorus and stroke risk, but none in China. To investigate the association between dietary calcium, magnesium, phosphorus, and stroke incidence in Chinese adults, we collected data from the China Health and Nutrition Survey (CHNS) from 2004 to 2011, including 6411 participants aged 45–79 years and free of stroke at baseline. Diet was assessed by interviews combining 3-d 24-h food recalls and household food inventory weighing at each survey round. The stroke incident was identified based on the validated self-report. Multivariate Cox regression models were used to estimate hazard ratios (HRs) and 95% confidence intervals (CIs). For 32,024 person-years of follow-up, 179 stroke cases were documented. After adjustment for major lifestyle and dietary risk factors, calcium intake was positively associated with reduced stroke risk, and the HR of stroke comparing extreme quartiles was 0.53 (95% CI 0.29–0.96, Ptrend = 0.03). In further stratified analyses, significant heterogeneity across sex strata was found (Pinteraction = 0.03). Dietary calcium intake among men was more inversely related to stroke, with HRs being 0.33 (95% CI 0.15–0.76, Ptrend = 0.02), compared to 1.24 (95% CI 0.46–3.35, Ptrend = 0.89) among women. However, no significant association between stroke and magnesium or phosphorus was revealed. Our findings suggest that higher dietary calcium intake was associated with a lower risk of stroke in Chinese adults, particularly in men.