A damage identification methodology considering shear deformation was presented in this paper to identify damage to semi-rigid frames with slender beams. On the basis of the successful identification of structural joint damage, the combined joint and member damage of the structure was identified. The objective function was formulated to minimize the discrepancies between the analytical and measured nodal displacements. Damage identification was performed on semi-rigid frame structures with different cross-sectional shapes, and the results were compared with those of ignoring shear deformation. Several frame structures were employed to verify the advantages and efficiency of the proposed method. The results demonstrate that the present method could significantly improve the accuracy of damage identification for semi-rigid frames compared with the method ignoring shear deformation.