With the rapid increase in demand for high-capacity optical communication networks (OCNs), driven by the global transition to digital platforms and the growth of online services after COVID-19, the issue of fiber nonlinearity, specifically four-wave mixing (FWM), becomes more important in long-distance OCN transmissions with multiple channels. This paper presents a new method that utilizes machine learning, specifically a convolution neural network (CNN), to mitigate the nonlinear impacts of FWM on the integrity of signals. We provide a thorough examination and simulation of the effects of FWM by utilizing a sophisticated CNN-based digital signal processing (DSP) model. Our analysis showcases the approach’s effectiveness in improving the quality of transmission. The CNN-based DSP model efficiently reduces nonlinear distortions caused by FWM and greatly optimizes the Bit Error Rate (BER) in extended OCN systems. Comparative evaluations demonstrate that our method decreases power penalties and improves the power budget by 1 dBm and 2 dBm, respectively, surpassing conventional compensating strategies. The simulation results continuously demonstrate BERs that remain below the critical threshold, confirming the enhanced transmission capabilities of the model. The experimental verifications of the CNN equalizer demonstrate its effectiveness in reducing crosstalk and improving power efficiency. The accuracy of our analytical models has been verified, demonstrating the capacity of the DSP model to produce the highest quality of transmission (QoT) with less required launch power to reach the target BER. The results strongly support the incorporation of our CNN-based DSP model into OCNs. This integration shows great potential for improving the management of multi-channel, long-distance, and high-data-rate optical transmissions in real-world OCN systems.