A model-free damage identification method for grid structures based on displacement difference is proposed. The inherent relationship between the displacement difference and the position of structural damage was deduced in detail by the Sherman–Morrison–Woodbury formula, and the basic principle of damage localization of the grid structure was obtained. That is, except for the tensile and compressive deformations of the damaged elements, the deformations of other elements were small, and only rigid body displacements occurred before and after the structural damage. According to this rule, a method for identifying the position of the damage was proposed for the space grid structure by using the rate of change of length for each element. Taking a space grid structure with a large number of elements as an example, the elastic modulus reduction method was used to simulate the damage to the elements, and the static and dynamic test parameters were simulated respectively to obtain the difference in displacement before and after the structural damage. The rate of change of length of each element was calculated based on the obtained displacement difference, and data noise was added to the simulation. The results indicated that the element with the larger length change rate in the structure was the most likely to be damaged, and the damaged element can be accurately evaluated even in the presence of noise in data.