With population aging, diabetes mellitus and cognitive function decline are common health problems among older adults worldwide. This longitudinal study is aimed at estimating the longitudinal associations of newly diagnosed prediabetes and diabetes status with cognitive function among Chinese adults aged 45 years and older and evaluating the clinical risk factors associated with cognitive function. Data were obtained from the China Health and Retirement Longitudinal Study (CHARLS). A total of 8716 participants meeting the inclusion criteria were enrolled between 2011 and 2012 at baseline, and 6125 participants completed the follow-up survey in 2018. Cognitive function, newly diagnosed diabetic status, depression, body mass index, and clinical and biochemical measurements were collected. At baseline, the mean age of the participants was 58.93 (SD: 9.76) years, 3987 (45.7%) were males, 1802 (20.7%) participants were newly diagnosed with prediabetes, and 935 (10.7%) were diabetes patients. After adjusting for control variables, diabetes was a significant risk factor for subsequent cognitive decline (unstandardized β estimate = − 0.50 , 95 % CI = − 0.98 ~ − 0.02 ). Subgroup analyses found that the association of diabetes with cognitive decline was significant in females. Stratification analyses found that among prediabetes patients, triglyceride concentrations were negatively associated with cognitive function; among diabetes patients, high-sensitivity C-reactive protein was significantly associated with cognitive decline. The newly diagnosed diabetes status at baseline was associated with subsequent cognitive decline among middle-aged and elderly Chinese, especially in females. The management of triglycerides through lifestyle modification for prediabetes and specific adjunctive anti-inflammatory therapy for diabetes might benefit cognitive performance.