In this study, an annular jet pump optimization method is proposed based on an RBF(Radial Basis Function) neural network model and NSGA-II(Non-Dominated Sorting Genetic Algorithm) optimization algorithm to improve the hydraulic performance of the annular jet pump applied in submarine trenching and dredging. Suction angle, diffusion angle, area ratio and flow ratio were selected as design variables. The computational fluid dynamics (CFD) model was used for numerical simulation to obtain the corresponding performance, and an accurate RBF neural network approximate model was established. Finally, the NSGA-II algorithm was selected to carry out multi-objective optimization and obtain the optimal design variable combination. The results show that the determination coefficient R2 of the two objective functions (jet pump efficiency and head ratio) of the approximate model of the RBF neural network were greater than 0.97. Compared with the original model, the optimized model's suction angle increased, and the diffusion angle, flow ratio and area ratio decreased. In terms of performance, the head ratio increased by 30.46% after the optimization of the jet pump, and efficiency increased slightly. The proposed jet pump performance optimization method provides a reference for improving the performance of other pumps.