Accurate medium- and long-term runoff prediction models play crucial guiding roles in regional water resources planning and management. However, due to the significant variation in and limited amount of annual runoff sequence samples, it is difficult for the conventional machine learning models to capture its features, resulting in inadequate prediction accuracy. In response to the difficulties in leveraging the advantages of machine learning models and limited prediction accuracy in annual runoff forecasting, firstly, the variational mode decomposition (VMD) method is adopted to decompose the annual runoff series into multiple intrinsic mode function (IMF) components and residual sequences, and the spectral clustering (SC) algorithm is applied to classify and reconstruct each IMF. Secondly, an annual runoff prediction model based on the adaptive particle swarm optimization–long short-term memory network (APSO-LSTM) model is constructed. Finally, with the basis of the APSO-LSTM model, the decomposed and clustered IMFs are predicted separately, and the predicted results are integrated to obtain the ultimate annual runoff forecast results. By decomposing and clustering the annual runoff series, the non-stationarity and complexity of the series have been reduced effectively, and the endpoint effect of modal decomposition has been effectively suppressed. Ultimately, the expected improvement in the prediction accuracy of the annual runoff series based on machine learning models is achieved. Four hydrological stations along the upper reaches of the Fen River in Shanxi Province, China, are studied utilizing the method proposed in this paper, and the results are compared with those obtained from other methods. The results show that the method proposed in this article is significantly superior to other methods. Compared with the APSO-LSTM model and the APSO-LSTM model based on processed annual runoff sequences by single VMD or Wavelet Packet Decomposition (WPD), the method proposed in this paper reduces the RMSE by 40.95–80.28%, 25.26–57.04%, and 15.49–40.14%, and the MAE by 24.46–80.53%, 16.50–59.30%, and 16.58–41.80%, in annual runoff prediction, respectively. The research has important reference significance for annual runoff prediction and hydrological prediction in areas with data scarcity.
