Current modeling methods of machine tool feed error are challenging to meet the demand of high-precision machining when facing complex machining conditions. To enhance the model’s predictive accuracy and the effectiveness of actual compensation, the Whale Particle Swarm Optimization (WPSO) algorithm is proposed to optimize the Backpropagation Neural Network (BPNN). Subsequently, the optimized network incorporates screw elongation and feed position as inputs to establish a feed-error prediction model. Ultimately, the established model was compared with other models and applied to real-time compensation experiments. The research results show that the proposed prediction model outperforms the BPNN model, the particle swarm-optimized BPNN model, and the whale-optimized BPNN model in various indicators. The accuracy of the prediction model was 93.12%, and the errors ranged from −3.80 μm to 4.57 μm with an average error of −0.30 μm. Under different operating conditions, the maximum backward and forward errors are reduced by 33.21% and 87.21%, and the average backward and forward errors are reduced by 57.15% and 84.37%, respectively. The error range is reduced by 67.41%. Beyond elevating prediction accuracy and compensation efficacy, the proposed model offers robust theoretical guidance for practical production.