Objectives Physical frailty is geriatric syndrome and can increase the risk of adverse outcome in the older population. Heart diseases are associated with physical frailty but a few studies in community-dwelling old population. Therefore the aim of this study was to examine the association between heart diseases, social factors, and physical frailty in community-dwelling older populations including the oldest old people. Methods The cross-sectional study included 1882 participants in community-dwelling older and oldest-old people in three age groups: 73 (±1), 83 (±1), and 93(±1) from both urban and rural areas of Japan. Questionnaires on medical history, psycho-social factors, blood samples, physical examinations, the hand grip strength test, and gait speed were measured at the venue. Physical frailty was defined based on a slow gait speed or weak grip strength. Heart diseases were assessed by self-reported questionnaires. Social interaction was based on the frequency of going outdoors and direct social contact. Analyses were conducted mainly using multiple logistic regression with adjustments for physical frailty risk factors. Results Heart disease subjects had a higher prevalence of physical frailty than those without heart disease (slow gait speed: 69.6 vs. 56.2%, p<.001; slow gait speed or weak grip strength: 80 vs. 69.6%, p=.002). After adjusting the covariate factors, heart diseases were associated with a slow gait speed (OR=1.5; 95%CI: 1.03-2.20, p=.035). Social interaction was associated with a slow gait speed (frequency of going outdoors: OR=0.87, 95%CI 0.79-0.9, p=.008; frequency of direct social contact: OR=0.88, 95%CI 0.82-0.95, p=.001), a weak grip strength (frequency of going outdoors: OR=0.86, 95%CI 0.77-0.96, p=.005), and with physical frailty (frequency of going outdoors: OR=0.84, 95%CI 0.75-0.94, p=.002; frequency of direct social contact: OR=0.89, 95%CI 0.82-0.97, p=.007). Living alone and frequency of direct social contact were associated with physical frailty among heart disease patients. Conclusions Our findings indicate that in community-dwelling older people, heart diseases and social interaction were associated with physical frailty. Older people with heart disease, those living alone and the frequency of direct social contact were associated with physical frailty. Future research must involve a longitudinal study to clarify the causal relationship.