A coal mine hoisting system includes two parts, one is a constant-length cable system, and the other is a variable-length cable system. In this paper, the nonlinear dynamic modeling of a coal mine hoisting system is established through Hamilton’s principle. The nonlinear partial differential equations of the coal mine hoisting system are discretized into ordinary differential equations by the fourth-order Galerkin truncation. The nonlinear dynamic responses and four key kinematic and structural parameter analysis of the coal mine hoisting system in the acceleration phases, constant velocity phases, and deceleration phases are given. The results show that the axial vibration displacements of the constant-length cable are an order of magnitude smaller than that of the variable-length cable. The load has the greatest effect on the axial vibration displacement of the hoisting cable. Adversely, the speed has the least effect on the axial vibration displacement of the hoisting cable.