This paper describes a class of nonlinear phenomena existing in the hypersonic flow and supersonic combustion process of scramjet engines: catastrophe, hysteresis and bifurcation, and further finds out the general rules (topological invariance) for the stability boundaries of mode transition in scramjet engines. With this topological invariance, a topological approach is put forward to model the stability boundaries, which may contribute to a complexity reduction of high-dimensional modeling when considering more perturbation parameters, and help to explore the physical laws of the nonlinear phenomena. Accordingly, this paper interprets the characteristic of combustion mode transition based on the cusp topological model in singular theories, and observes the bifurcation characteristic in combustion mode transition. Moreover, a modeling approach is proposed to mathematically describe the stability boundaries of combustion mode transition in scramjet engines, and the model has high accuracy comparing to the simulation data, which proves the validation of the basic ideas proposed in this paper. Finally, future research directions are proposed. scramjet engine, catastrophe, bifurcation, model