Objective: Suicide is a major public health and social problem in contemporary societies. Previous studies showed that the older the seniors were, the more likely it was that they would experience disability, chronic disease, or both. The objective of this study was to examine the joint effects of chronic disease and physical disability on suicide ideation while controlling for psychological distress among the rural elderly living in Shandong Province, China. Method: A total of 5514 rural elderly individuals (60+) living in Shandong Province, China were included in this study. Suicidal ideation was assessed by using questions from the National Comorbidity Survey (NCS). Multiple logistic analyses were performed to examine the factors associated with suicide ideation. A path analysis was conducted to test the direct and indirect effects of chronic disease and of activity of daily living (ADL) limitation on suicide ideation while controlling for psychological distress. Results: The prevalence of suicide ideation among the rural elderly in Shandong, China was 11.0%. Psychological distress had the strongest direct (β = 0.392) and total effect (β = 0.392), chronic disease (β = −0.034; β = −0.063) had both direct and indirect impacts, and ADL (β = 0.091) had indirect impacts on suicide ideation. Psychological distress was a mediator between chronic disease, ADL limitation, and suicide ideation. Conclusions: Psychological distress was the greatest influencing factor of suicide ideation among the rural elderly, followed by chronic disease and disability. Effective intervention measures should be taken to facilitate the early detection of psychological distress in clinical practice among the rural elderly.