According to a generalized Hamilton’s principle, three-dimensional (3D) nonlinear vibration equations for overhead transmission lines that consider geometric nonlinearity are established. Based on the characteristics of an actual transmission line, the 3D equations are simplified to two-dimensional equations, and the nonlinear vibration behavior of transmission lines is investigated by combining theoretical analysis with numerical simulation. The results show that transmission lines have inherently nonlinear vibration characteristics. When in free vibration, a transmission line can undergo nonlinear internal resonance, even when its initial out-of-plane energy is relatively low; as its initial out-of-plane energy increases, the coupling of in-plane and out-of-plane vibration becomes stronger. When forced to vibrate by an external excitation, due to the combined action of internal and primary resonance, the vibration energy of a transmission line transfers from the out-of-plane direction to the in-plane direction that is not directly under the excitation, resulting in an increase in the dynamic tension and the displacement amplitude of the transmission line. Increasing damping can consume the vibration energy of a transmission line but cannot prevent the occurrence of internal resonance.