Diabetes is on the rise as the worldwide population ages. While physical activity can help protect against diabetes, ageing is commonly associated with reduced physical activity. This study aimed to examine if physical activity differs by diabetes status in mid-aged adults, how this association changes over time, and whether physical activity-related sociodemographic factors and health indicators differ in those with and without diabetes. Data came from four waves of the How Areas in Brisbane Influence HealTh and AcTivity (HABITAT), a longitudinal study of mid-age adults living in Brisbane, Australia. Random effects/Expectation-maximisation (RE-EM) regression trees were used to identify factors affecting physical activity among those with and without diabetes, both separately and combined. At study entry, those with diabetes had a higher median age of 58 years (95% CI: 57–60) and a lower median physical activity of 699 MET.min/week (95% CI: 599–799) than people without diabetes (53 years (95% CI: 53–53) and 849 MET.min/week (95% CI: 799–899)). However, the strongest factors influencing physical activity were BMI and gender, not diabetes status. It is vital to promote physical activity among adults, in particular among those with high BMI and women, as well as those with and at high risk of diseases like diabetes.