Firstly, by analyzing the response of Charpy V-notch specimen impacted by pendulum, the relationship between specimen geometry, material properties, and impact energy is established and simplified, and the mathematical model for evaluating impact energy of specimens with different sizes is established. Then, the effectiveness of the model through a series of impact tests is verified. Theoretical analysis and experimental results show that the relationship between ligament length and impact energy is quadratic, while the relationship between ligament thickness and impact energy is linear. In the derivation process, the intrinsic impact toughness is used to evaluate the toughness of materials. The mathematical model makes it possible to evaluate the impact energy of specimens with different sizes and provides a theoretical basis for evaluating the impact resistance of structures.