Polarization variations in optical fibers are complex and would severely affect the performances of polarizationsensitive signal distribution systems. Owing to advances in experimental techniques and theoretical tools, we observe the effect of polarization variations in the optical fibers and we demonstrate that polarization mode dispersion (PMD) dominates the free-running fiber noise. Ultrahigh correlation, over 99%, is found between the phase fluctuation induced by polarization variation and the temperature fluctuation impacted on the optical fibers. By means of a polarization scrambling technique, the polarization variations in the optical fiber are averaged and the phase perturbations are minimized. It provides a promising solution for suppressing the random disturbances of polarization in the fiber-based optical frequency transfer system and quantum key distribution (QKD) system.