Van der Waals heterostructures built by vertically stacked transition metal dichalcogenides (TMDs) exhibit a rich energy landscape including interlayer and intervalley excitons. Recent experiments demonstrated an ultrafast charge transfer in TMD heterostructures. However, the nature of the charge transfer process has remained elusive. Based on a microscopic and material-realistic exciton theory, we reveal that phonon-mediated scattering via strongly hybridized intervalley excitons governs the charge transfer process that occurs on a sub-100fs timescale. We track the time-, momentum-, and energy-resolved relaxation dynamics of optically excited excitons and determine the temperature-and stacking-dependent charge transfer time for different TMD bilayers. The provided insights present a major step in microscopic understanding of the technologically important charge transfer process in van der Waals heterostructures.