Compact binaries are within the reach of gravitational and electromagnetic wave detectors, and are important for our understanding of astrophysical environments and the composition of compact objects. There is a vast body of work devoted to the evolution of such binaries in background media, such as in common-envelope evolution, accretion disks and dark matter mini-spikes. Here, we explore further gravitationally-bound binaries evolving within an environment. We show that dissipative effects such as gravitational drag and accretion impart a momentum to the center of mass of asymmetric binaries. We numerically evolve the binaries in a Newtonian setup and show that, depending on the medium density, the center of mass can accelerate to high speeds -in some cases 300 km/s or more -during inspiral, with potentially observable signatures. Our numerical results are fully consistent with an analytical result for the CM evolution at first order in the medium density.