THz radiation promises breakthrough advances in compact advanced accelerators due to the high frequency and GV/m fields achievable, orders of magnitude larger than in conventional radiofrequency (RF) based accelerators. Compared to laser-driven schemes, the large phase acceptances of THz-driven accelerators are advantageous for operation with sizable charge. Despite burgeoning research, THz sources, particularly laser-based ones, cannot yet compete with the efficiency of RF amplifiers for high average current accelerators. Nevertheless, THz-based phase space manipulation is of immediate interest for a variety of applications, including generation and diagnostics of ultrashort bunches for electron diffraction/microscopy and compact free-electron laser applications.The challenge of maintaining overlap and synchronism between an electron beam and short laser-generated THz pulse has so far limited interactions to the few mm scale. We discuss a novel scheme for simultaneous group and phase velocity matching of nearly single-cycle THz radiation with a relativistic electron beam for meter-scale interaction. We demonstrate energy modulations of up to 150 keV using modest THz pulse energies (≤ 1 µJ). We apply this large and efficient energy exchange for beam compression and time-stamping of a relativistic beam, paving the way towards realizing the unique opportunities enabled by laser-based THz accelerators.