Cemented gangue backfill material (CGBM) is made of coal gangue, fly ash, cement, and water. The backfill body will be corroded by mine water that contains different kinds of corrosive ions. The paper mainly aimed to study the diffusion process of sulfate ion (SO42−) in CGBM. Soluble salt solutions with different concentrations are prepared to simulate various mine water. The SO42− concentration was measured by layered sampling method and chemical analyzing method. The diffusion coefficient was calculated according to the second Fick’s law. Scanning electron microscope (SEM) and X-ray diffraction (XRD) were used to analyze the degradation mechanisms. The experimental results showed that the SO42− concentration increases in CGBM when the external sulfate solutions concentration increases. With the increasing of magnesium ion (Mg2+) concentration and bicarbonate ion (HCO3−) concentration in the external corrosion solutions, the diffusion coefficient of SO42− was increased, and HCO3− has more significant effects than Mg2+. SO42− reacting with CGBM produced ettringite and gypsum. The ettringite and gypsum changed the pore structure of hardened CGBM and affected the compressive strength of hardened CGBM. Studying the diffusion of SO42− can contribute to predicting the service life of CGBM.