Heterotrophic Bacteria and Archaea (prokaryotes) are a major component of marine food webs and global biogeochemical cycles. Yet, there is limited understanding about how prokaryotes vary across global environmental gradients, and how they may be affected by climate change. Here, using parametric statistical models fit to a global dataset of prokaryotic abundance and cell carbon, we find that prokaryotic biomass varies by only 2-fold across the global ocean’s surface waters, despite phytoplankton varying by >2 orders of magnitude, while prokaryote abundance increases with temperature by 2.6% per °C. As a result, in contrast to the 10% projected decline in global biomass of larger heterotrophs (zooplankton and fish), prokaryote biomass will decline only 3% by 2100 under climate change. Thus, future oceans could be increasingly dominated by heterotrophic prokaryotes, diverting a larger share of primary production away from larger heterotrophs and into microbial food webs.