The combination of non-trivial band topology and symmetry breaking phases gives rise to novel quantum states and phenomena such as topological superconductivity, quantum anomalous Hall effect and axion electrodynamics. Evidence of intertwined charge density wave (CDW) and superconducting order parameters has recently been observed in a novel kagome material AV3Sb5 (A=K,Rb,Cs) that features a Z2 topological invariant in the electronic structure. However, the origin of the CDW and its intricate interplay with topological state has yet to be determined. Here, using hard x-ray scattering, we demonstrate a three-dimensional (3D) CDW with 2 × 2 × 2 superstructure in (Rb,Cs)V3Sb5. Unexpectedly, we find that the CDW fails to induce acoustic phonon anomalies at the CDW wavevector but yields a novel Raman mode that quickly damps into a broad continuum below the CDW transition temperature. Our observations exclude strong electron-phonon coupling driven CDW in AV3Sb5 and point to an unconventional particle-hole condensation mechanism that couples CDW, superconductivity and topological band structure.

The Kitaev quantum spin liquid epitomizes an entangled topological state, for which two flavors of fractionalized low-energy excitations are predicted: the itinerant Majorana fermion and the Z2 gauge flux. It was proposed recently that fingerprints of fractional excitations are encoded in the phonon spectra of Kitaev quantum spin liquids through a novel fractional-excitation-phonon coupling. Here, we detect anomalous phonon effects in α-RuCl3 using inelastic X-ray scattering with meV resolution. At high temperature, we discover interlaced optical phonons intercepting a transverse acoustic phonon between 3 and 7 meV. Upon decreasing temperature, the optical phonons display a large intensity enhancement near the Kitaev energy, JK~8 meV, that coincides with a giant acoustic phonon softening near the Z2 gauge flux energy scale. These phonon anomalies signify the coupling of phonon and Kitaev magnetic excitations in α-RuCl3 and demonstrates a proof-of-principle method to detect anomalous excitations in topological quantum materials.