Heavy metal pollution of soil in agricultural areas is the most prominent environmental pollution problem in China, seriously affecting human health and food security. It has become one of the environmental problems to which all sectors of society attach great importance. Soil heavy metals in the weathering area of hazardous geological bodies in southwest China have naturally high background attributes. Therefore, ecological risk assessment and analysis of potential sources of soil heavy metals in southwest China is of great significance for soil health management, soil heavy metal pollution control and territorial spatial planning. In this study, we collected 787 soil samples (0–20 cm) in Xuanwei County in China and analyzed the concentrations of As, Cd, Cr, Cu, Hg, Ni, Pb and Zn in soils. Igeo, RI, HI and CR were used to calculate the pollution levels, ecological risks and human health risks. Additionally, the PMF model and one-way ANOVA were used to identify the potential sources and discuss the factors affecting the enrichment of heavy metals. The results showed that the mean contents of the surface soils were 1.190 (Cd), 139.4 (Cr), 96.74 (Cu), 0.081 (Hg), 56.97 (Ni), 46.66 (Pb) and 130.1 (Zn) mg/kg. All heavy metals exceeded the background values of the A layer soil in Yunnan Province. The Igeo showed that Cd was the most hazardous element in the study area, followed by Cu, Cr, As, Ni and Pb. The RI showed that low ecological risks, moderate ecological risks, considerable ecological risks and strong ecological risks accounted for 3.81%, 55.27%, 37.74% and 3.18%, respectively, of the total samples, and Cd was the main dominant element. The HI values of the As element in children were greater than 1, indicating a non-carcinogenic risk, and other elements’ risks were acceptable. The CR values of Cr and Ni were higher than their limits (1 × 10−4), and both had carcinogenic risks in children and adults, as did As in children. According to the PMF model, four heavy metals sources were identified: geological sources (32%), sources from mining activities (19.38%), atmospheric deposition sources (17.57%) and agricultural sources (31.05%). Thereinto, As and Pb were mainly derived from agricultural sources, Cd and Cr were mainly associated with geological sources, Cu was largely from mining activity sources, Hg was mainly from atmospheric deposition sources and Ni and Zn were mainly from geological sources, mining activities and agricultural activities. The parent material has a significant influence on the enrichment of heavy metals in the soil, and the heavy metals are significantly enriched in the carbonate parent material and quaternary parent material. Topography also plays a role in heavy metal accumulation; Cd, Cr, Cu, Ni and Zn gradually decreased with the increase in altitude, and As and Pb increased with the increase in altitude. Mn-oxide played a crucial part in the enrichment of Cu and Zn, while SOC, K2O and pH had little influence on the accumulation of heavy metals.