In the gear shifting process of a clutchless automated mechanical transmission (AMT), the potential collision between sleeve and dog gear may increase both vehicle jerk and gear changing duration. Therefore, the gear shifting process's mechanism analysis and dynamic modeling are the essential prerequisites to improve gear shifting quality. This paper analyzes the potential collision between the sleeve and the dog gear which both with concave-convex profiles, the tooth-side chamfer considered. To model the collision process, a fast-determining collision face method based on an intersection map is proposed, and the Monte Carlo method and equivalent depth are employed to calculate the collision stiffness. Then, the stiffness-damping model of the collision process and an 8-degree-of-freedom dynamic model of the gear-shifting system are established. Finally, taking a two-speed clutchless and non-synchronizer automated mechanical transmission (NSAMT) as an example, the established dynamic model of the gear-shifting system is applied, and simulation verification is performed in Matlab/Simulink. The simulation results show that the established model can accurately reflect the changes of forces, speeds, and displacements of sleeve and dog gears during the collision and effectively reflect processes of the disengagement of the sleeve, the speed changes in active synchronization, the engagement of sleeve and gear keeping. Besides, the results also show that the tooth-side chamfer will increase the disengagement time, while it is conducive to gear keeping, and that unloading the drive motor can decrease the disengagement time due to the existence of the tooth-side chamfer.