High-energy photons with O(MeV) energies from radioactive contaminants can scatter in a solidstate target material and constitute an important low-energy background for sub-GeV dark matter direct-detection searches. This background is most noticeable for energy deposits in the 1−100 meV range due to the partially coherent scattering enhancement in the forward scattering direction. We comprehensively quantify the resulting single-and multi-phonon background in Si, Ge, GaAs, SiC, and Al2O3 target materials, which are representative of target materials of interest in low-mass dark matter searches. We use a realistic representation of the high-energy photon background, and contrast the expected background phonon spectrum with the expected dark matter signal phonon spectrum. An active veto is needed to suppress this background sufficiently in order to allow for the detection of a dark matter signal, even in well-shielded environments.