Sulfate erosion is a threat to durability and sustainability of pavement structure. In this work, the performance degradation of asphalt binder under internal sulfate erosion was investigated. Different dosages of Na2SO4 (0 wt.%, 2.5 wt.%, 5 wt.%, and 10 wt.%) samples were prepared to investigate the effect of sulfate on the performance of the asphalt binder. The surface tension test and low-temperature rheological property test were carried out to evaluate the adhesion of sulfate-incorporated asphalt mastic. Rapid freezing and thawing test with different concentrations of sulfate solution was conducted to explore the effect of sulfate concentrations and freeze-thaw cycles on the softening point and force ductility of matrix asphalt. Phases of the asphalt binder were characterized by Fourier transform infrared (FTIR) spectroscopy. The results show that sulfate has a negative impact on the road performance of the asphalt binder. Internal sulfate erosion decreased the adhesion of the asphalt binder. Also, the low-temperature rheological property of asphalt binder was deteriorated. After the freeze-thaw cycles of external sulfate erosion, softening point and tensile force peak of matrix asphalt increased and the low-temperature ductility decreased. The main reason of performance deterioration of the asphalt binder is the “salt aging” effect causing by sulfate erosion. The research results can provide useful reference for the durability design of asphalt mixture in sulfate-enriched regions.