The stability of steady-state synchronous and nonsynchronous response of a nonlinear rotor system supported by squeeze-film dampers is investigated. The nonlinear differential equations that govern the motion of rotor bearing systems are obtained by using the Generalized Polynomial Expansion Method. The steady-state response of the system is obtained by using the hybrid numerical method, which combines the merits of the harmonic balance and collocation methods. The stability of system response is examined using the Floquet-Liapunov theory. Using the theory, the performance may be evaluated with the calculation of derivatives of nonlinear hydrodynamic forces of the squeeze-film damper with respect to displacement and velocity of the journal center. In some cases, these derivatives can be expressed in closed form and the prediction of the dynamic characteristic of the nonlinear rotor system will be more effective. The stability results are compared to those using a direct numerical integration method and both are in good agreement.