To verify the damping improvement by replacing partial sand with rubber powder in the concrete process, this study investigated the effects of the rubber powder (5%, 10%, 15%, and 20%) on the mechanical properties and micromechanism of polyvinyl alcohol (PVA) fiber-reinforced concrete. In addition, in order to discuss its damping performance, free vibration test was conducted. Microstructure analyses were conducted by the scanning electron microscope (SEM) test. These results indicated that, as the content of the rubber powder increased, the damping ratio increased, and the compressive strength decreased, but this strength loss can be effectively controlled by adding PVA fiber. The rubber powder with a volume content of 5% and PVA with a mass of 2.4 kg/m3 were the most optimal mixing to balance the strength and damping requirement. According to the SEM test results, the rubber powder was beneficial to improve the damping ratio of PVA concrete, but it aggravated its interface defects.