In order to ensure the accuracy of numerical simulation results for the Gill-piece separation device (GPSD), an exploration of the optimal mathematical model and mesh parameters was conducted. The Mixture model in CFX was coupled with RNG k-ɛ, SST, BSL, and SSG turbulence models to simulate the water-sand two-phase flow field in the GPSD under dynamic water conditions. By comparing the numerical simulation results with physical experimental phenomena, it is found that the velocity vector diagram calculated by the Mixture-RNG k-ɛ coupling model conforms more closely to the physical experimental phenomena (Double-layered Counterflow), and the relative error of the water-sand separation efficiency calculated by the Mixture-RNG k-ɛ coupling model is very small, only 1.77%. Thus, it can be regarded as the optimal mathematical model for numerical simulation of the GPSD under dynamic water conditions. Considering factors such as computational time, the number of meshin numerical simulation should be set to around 300,000 for the best performance.