With the improvement of the complexity and reliability of mechanical equipment, it has been difficult for the commonly used variational modal decomposition method of vibration signal of rotating machinery to meet the current practical engineering requirements. In order to further improve the adaptability, processing efficiency, and robustness of rotating machinery fault diagnosis methods, a collaborative hybrid element heuristic to multiobjective optimization algorithm is introduced in this paper. Combined with variational modal decomposition (VMD) method, the fault diagnosis method of rolling bearing under complex working conditions is studied. This paper mainly uses a collaborative hybrid metaheuristic algorithm to improve the nondominated sorting genetic algorithm II (NSGA II) and multiobjective particle swarm optimization (MOPSO), which improves the convergence efficiency of multiobjective optimization method and solves the problem of uneven distribution of optimal solutions. Then, the improved multiobjective optimization algorithm is combined with VMD to solve the problem of parameter selection of the VMD method under complex working conditions of rotating machinery. At the same time, the variation relationship between various signal features and VMD decomposition results is compared and studied, and the features with good effect are taken as the objective function of the optimization algorithm; the ability of denoising and feature extraction of VMD in rotating machinery fault diagnosis is improved. In this paper, the proposed method is explored by using analog signals and experimental data of rolling bearings. Through comparison, the improvement of adaptive ability, operation speed, and robustness of the proposed method in rotating machinery fault diagnosis is verified.