Chattering in the impact damper refers to a continuous impact phenomenon during vibration. This paper establishes the dynamic model of the nonfixed constrained vibro-impact system on the base of classical impact damper. The chattering occurrence in the low frequency range is studied by numerical simulation. The results show the frequency ratio and clearance significantly affect the system's chatter behavior: (1) Chattering is prone to occur in the low frequency range. As the frequency ratio decreases, the number of unilateral continuous impacts between the primary system and the small ball (the free mass) increases from one impact to multiple impacts, and a viscosity behavior appears under the condition of low frequency ratio. (2) Small clearance is prone to chattering. With the increase of the clearance, the frequency ratio range of the chattering in the system decreases. When the clearance exceeds a certain critical value, the system will not experience chattering behavior.