Background: Diabetes mellitus (DM) is a recognised risk factor for cognitive dysfunction. The purpose of this study was to explore the relationship between active treatment for DM and cognitive function in middle-aged (< 60 years) and older adults (≥60 years), respectively.Methods: A total of 13,691 participants (58.55 ± 9.64 years, 47.40% of men) from the Chinese Health and Retirement Longitudinal Study (CHARLS) were included. The participants were classified into three groups according to whether or not they have diabetes and to their diabetes treatment status: diabetes-free, treated-diabetes and untreated-diabetes, in which the diabetes-free group was regarded as reference specially. Cognitive function was assessed by two interview-based measurements for mental intactness and episodic memory.Results: Compared with the participants in the diabetes-free group, the older participants in the treated-diabetes group had better performance in terms of mental intactness (β = 0.37, 95% CI = 0.04–0.70). No significant association was observed in the middle-aged participants. In the subgroup analyses, the lower cognitive score was only observed in people without depression, who had never smoked and drunk, and with a normal weight (body mass index: 18.5–23.9 kg/m2).Conclusion: The cognitive function of actively treated diabetic patients was better than that of patients without diabetes, but the improvement was significant only in elderly people. Depression, smoking, drinking, and an abnormal weight may attenuate this effect.