Two major challenges associated with avibration-based damage detection method using changes in natural frequencies are addressed: accurate modeling of structures and the development of a robust inverse algorithm to detect damage, which are defined as the forward and inverse problems, respectively. To resolve the forward problem, new physics-based finite element modeling techniques are developed for fillets in thin-walled beams and for bolted joints, so that complex structures can be accurately modeled with a reasonable model size. To resolve the inverse problem, a logistic function transformation is introduced to convert the constrained optimization problem to an unconstrained one, and a robust iterative algorithm using the Levenberg-Marquardt method is developed to accurately detect the locations and extent of damage. The new methodology can ensure global convergence of the iterative algorithm in solving under--determined system equations and deal with damage detection problems with relatively large modeling error and measurement noise. It is applied to various engineering structures including lightning masts, a space frame structure and one of its components, and a pipeline. The exact locations and extent of damage can be detected in the numerical simulation, and the locations and extent of damage can be successfully detected in experimental damage detection.