The friction-induced vibration of a novel 5-DoF (degree-of-freedom) mass-on-oscillating-belt model considering multiple types of nonlinearities is studied. The first type of nonlinearity in the system is the nonlinear contact stiffness, the second is the non-smooth behaviour including stick, slip and separation, and the third is the geometrical nonlinearity brought about by the moving-load feature of the mass slider on the rigid belt. Both the linear stability of the system and the nonlinear steady-state responses are investigated, and rich dynamic behaviours of the system are revealed. The results of numerical study indicate the necessity of the transient dynamic analysis in the study of friction-induced-vibration problems as the linear stability analysis fails to detect the occurrence of self-excited vibration when two stable solutions coexist in the system. The bifurcation behaviour of the steady-state responses of the system versus some parameters is determined. Additionally, the significant effects of each type of nonlinearity on the linear stability and nonlinear steady-state responses of the system are discovered, which underlie the necessity to take multiple types of nonlinearities into account in the research of friction-induced vibration and noise.