In order to numerically simulate the whole fracture process including the initiation and propagation of crack in asphalt concrete semicircular specimens under external force, the extended finite element method (XFEM) was adopted considering the shortcomings of the conventional finite element method (FEM). The fracture processes of the semicircular specimens under 5 kinds of loading modes, Me, were analyzed, and the simulation results were compared to the actual fracture paths in the actual specimens. The results indicated that the critical effective stress intensity factor will decrease first and then increase with the increase of Me, and the XFEM simulation results are similar to that of the actual specimens in crack initiation angle and propagation path in the 5 different loading modes. It is proved that the XFEM is very effective in simulating the fracture process and has obvious advantages compared with the FEM. According to the stress state at the crack tip, the initiation angle and its propagation paths were analyzed, and it was pointed out that the increase of the shear stress component caused the crack initial angle to increase with the increase of Me.