Nonlinear dynamics and stability of the rotor-bearing-seal system are investigated both theoretically and experimentally. An experimental rotorbearing-seal device is designed and corresponding tests are carried out. The experimental rotor system is simplified as the Jeffcott rotor. The nonlinear oilfilm forces are obtained under the short bearing theory and Muszynska nonlinear seal force model is used. Numerical method is utilized to solve the nonlinear governing equations. Bifurcation diagrams, waterfall plots, Poincaré maps, spectrum plots and rotor orbits are drawn to analyze various nonlinear phenomena and system unstable processes. Theoretical results from numerical analysis are in good agreement with results from experiments. Conclusions are drawn and prove that this study will contribute to the further understanding of nonlinear dynamics and stability of the rotor system with the fluid-induced forces from oilfilm bearings and the seals.