The accurate prediction of train delays is a critical issue for ensuring safe and reliable train operations, as well as for providing passengers with up-to-date and reliable information. Despite an extensive amount of literature on train delay prediction models [1-3], the absence of a reliable standard evaluation framework raises critical questions as posed by [4]: "Are we confident that our models are better, in terms of accuracy, than models developed 30 years ago?" As both research and practice mature, thorough testing and evaluation of prediction algorithms is needed to establish a solid knowledge foundation for the development of future robust real-time train delay prediction algorithms. To address this challenge, this paper proposes a comprehensive evaluation framework for real-time train delay prediction that takes into account three different dimensions: overall, spatial, and temporal performances. The proposed framework aggregates data at various levels, enabling a more comprehensive assessment of prediction models' performance and facilitating the identification of prediction patterns. The study applies the evaluation framework to three different prediction algorithms and evaluates their performance across multiple Swedish train stations over a three-month period. The results demonstrate that the AI solution outperforms both the graph and manual methods. The graph method predicts train delays better on weekends but struggles during peak hours on weekdays. The AI solution exhibits stable performance in predicting train arrival delays, especially during peak hours, and with greater accuracy at busy stations. The study suggests that a combination of AI and graph methods can complement each other in predicting train arrival delays across different time periods and station types. The proposed evaluation framework can be used to assess the effectiveness of different prediction models and can facilitate the development of more reliable and accurate train delay prediction algorithms. This research can have important implications for improving the safety and reliability of train operations, as well as enhancing the overall passenger experience.Machine learning-based evaluation of risk of delay for the real-time Railway Traffic Management Problem