According to the general drive system consisting of motor, gear reducer and working-load, a torsion dynamics model with three-degree-of-freedom is established, in which the nonlinear resistance torque is considered. Based on the Hurwitz algorithm criterion, the Hopf bifurcation and critical points are analyzed and the stable parameter domains are obtained. Furthermore introducing the feedback controller into the initial system, the stability of self-excited vibration originating from the way of Hopf bifurcation is investigated. The critical parameters can be shifted by adjusting the linear gain for the purpose of enlarging the stable parameter domains. Applying the central manifold theorem, the amplitude of periodic motion and the unstable divergent vibration can be suppressed through selecting appropriate nonlinear control parameter. In addition, all these theoretical results are verified by numerical simulation.