In order to analyze the torsional shear process of asphalt mixtures in a microscopic view, the numerical simulation of a torsional shear test of an asphalt mixture was carried out by discrete element method. Based on the defects of existing algorithms, the method of random reconstruction of the existing 3D model of the asphalt mixture was improved, and a new reconstruction method was proposed. A 3D numerical model of the asphalt mixture contained irregular-shaped coarse aggregate, mineral gradation, and asphalt mortar; furthermore, the particle algorithm established the air void distribution. Then, the numerical simulation of the asphalt mixture’s torsional shear was completed; in addition, the stress, displacement, and contact of the specimens at each stage were analyzed. The results showed that the stress and displacement in different stages changed greatly with the loading, i.e., the crack generated from a weak point on the surface and then spread to the ends with an oblique angle of about 45°. At the same time, the shear failure process of the asphalt mixture was studied. The virtual test method could accomplish the implementation of the numerical simulation of torsional shear; it also provided a good research method for analysis of the asphalt mixture’s shear failure process.