Secondary flow transfers the boundary layer from endwall to suction side corner region, which deteriorates the performance of compressors. Vortex Generators (VGs) can attenuate the secondary flow when implemented at the endwall of blade passage. In order to make an efficient optimization design of the VGs layout, mathematical models are introduced to replace the actual VGs. However, the model is only effective at small incidence less than 20 degrees. In this paper, firstly, a mathematical model is presented to replace the actual VGs facing a lager incidence, and validated by CFD results with actual VGs. Secondly, an efficient design method for optimizing VGs layout is presented based on VG model and GA-SVM method. The method is illustrated in a highly loaded compressor stator, and the VGs layout is optimized and applied in the blade passage. The results show that the VGs can prevent the secondary flow from the suction corner, and reduce the total pressure loss coefficient at the trailing edge of the stator by 15%. Finally, the flow mechanism of the VGs is investigated using numerical method. The results show that the VGs can induce a primary vortex at the tip and prevent the circumferential migration of the boundary layers. Moreover, the secondary flow can produce a large incidence at the hub of the VGs, thus increasing the aerodynamic losses of the VGs secondary vortex.