Vitrimer-based composites combine the excellent performance of thermoset materials with the recyclability of thermoplastic materials. In this paper, based on ballistic impact experiments, the high-velocity impact resistance of CFRP before and after matrix modification and the failure mechanism of Vitrimer-based composites are investigated. The better impact resistance of Vitrimer-based composites is demonstrated in terms of ballistic limit, energy absorption, and failure modes. The failure mechanism of Vitrimer-based composites, which is different from that of conventional Epoxy-based composites, is found by CT scanning and verified in the strain signals.