Physical activity and health are of significant importance for the rapid aging population in China. Built environment has been suggested to be associated with elderly physical activity and health. However, the association differences between cities remain unclear. Perceived built environment scores and elderly leisure-time physical activity (LTPA) of 308 elderly in Hangzhou and 304 elderly in Wenzhou were collected using Neighborhood Environment Walkability Scale and International Physical Activity Questionnaire, respectively. A multivariate linear regression method and T-test were used to analyze of the associations between elderly LTPA and built environment and the differences between the two cities, respectively. The results showed that LTPA was positively associated with walking/cycling facilities and crime safety in both cities. LTPA was positively correlated with residential density, aesthetics, pedestrian/traffic safety in Wenzhou and negatively correlated with access to services in Hangzhou. The perceived scores of aesthetics (2.71 vs. 2.45) and pedestrian/traffic safety (2.11 vs. 1.71) in Hangzhou were significantly higher than those in Wenzhou. The results suggested that built environment elements like higher walking/cycling facilities and crime safety may motivate elderly engaging LTPA in both cities. However, LTPA was affected by different factors in these two cities. In the urban redevelopment, survey conducted in its own city would provide meaningful information and cannot be neglected.