Gear transmission is the most basic transmission component in mechanical transmission system. Many scholars have done a lot of research on gear reliability. When the variation coefficient is used to calculate and optimize the reliability of bevel gear, in order to calculate the reliability of bevel gear, it is often assumed that the gear works under constant torque, that is, the coefficient of variation (COV) is zero, but this is not the case in practice. In this paper, a gear reliability method based on discrete element simulation is proposed. The purpose of this method is to simulate the actual working conditions of gears, calculate more accurate coefficient of variation in the real world, and improve the accuracy of gear reliability design. Firstly, the real working conditions of the bevel gear transmission are simulated by discrete element method (DEM), and in the transmission system, the tangential force COV of the bevel gear is proved to be equal to the torque COV of the crusher central shaft. Secondly, the multi-objective function model of the gear transmission system is established based on the double tooth roll crusher (DTRC). The optimal volume and reliability of the bevel gear transmission are taken as the objective function, and the teeth number, module and face width factor of basic parameters of gear are optimized by genetic algorithm (GA). Finally, the accuracy of the optimization results is verified by Monte Carlo method. The main purpose of the manuscript is to analyse the effect of actual conditions (DEM simulation) on the optimization results. The results show that the COV of nominal tangential load of bevel gear is about 0.65 under actual working conditions, so in order to guarantee the same reliability, total volume need to be increased by 34.4%. This method is similar to the selection of gear safety factor. In practical production, the selection of safety factor is often based on experience. This paper provides a new method to optimize the reliability of bevel gear, combining with DEM simulation, which provides theoretical guidance for optimal design of bevel gear.