Soil contamination by potentially toxic trace elements (PTEs) such as Cadmium (Cd), is a major environmental concern because of its potential implications to human health. Cacao-based products have been identified as food sources with relatively high Cd contents. Here, we assessed Cd concentrations of cacao-growing soils in four major agricultural regions with contrasting climates in Peru, one of the main exporters of cacao products worldwide. At each study site (n = 40) a broad range of potential factors affecting Cd concentration in soils, i.e., site, soil and management, were evaluated. Concentrations of Cd ranged between 1.1–3.2 mg kg−1. Mean values per region were below 2.7 mg kg−1, usually established as upper-limit for non-polluted soils. Cadmium concentrations were significantly (p < 0.001) higher in sites at higher elevations and in a temperate, drier climate. Cadmium correlated positively with pH (r = 0.57; p < 0.05) and was higher (p < 0.001) in alluvial sediments and Leptosols. Management factors (cacao variety, cultivation year, management practices) and agroecology did not affect Cd concentrations directly. Overall, this study highlights the importance of considering a broad range of both natural and anthropogenic factors to evaluate Cd concentrations in cacao-growing soils and contribute to effective and sustainable cacao production by improving land management and planning.