The main retarder of vehicle is gear rotor system, its vibration property is complex. The whole transfer matrix method is a new algorithm to study, relationship of the displacements and forces are established considering gyroscopic moment and coupling vibration. The meshing point between rotors is treated as coupling element, balance equations are estalished on deformation compatibility condition. The globe status vectors including forces and displacements of all shafts were integrated into a whole transfer matrix. So it can be programming and has high numerical stability and accuracy. The influence of the meshing effect on vibration of the system can be systematically studied. The status vectors such as displacements, deflection angles, torsion angles, shearing forces, bending moments and torques are no longer independent. The vibration property of the geared rotor system was calculated and analyzed. The numerical results revealed that many coupling critical rational speeds of the system are derived. vibration of main retarder can be improved.