This paper uses Lagrangian statistics—absolute dispersion, pair dispersion, and particle forecasting—to compare four eddy‐permitting models of idealized mesoscale ocean dynamics. The baseline model uses scale‐selective biharmonic damping of momentum to keep the velocity field smooth at the grid scale without overly smearing the partially resolved eddies. The second model uses a nonlinear space‐ and time‐varying Leith viscosity to absorb enstrophy near the grid scale without dissipating too much energy. The last two models use a strong biharmonic damping of momentum to keep the velocity field smooth at the grid scale; this dissipates an unrealistically large amount of kinetic energy, and the models add either deterministic or stochastic forcing to reinject this energy at larger scales. These models are called Jansen and Held models after Jansen and Held (2014, https://doi.org/10.1016/j.ocemod.2014.06.002). All four models correctly model the diffusive transport of particles over time scales more than a few weeks, as measured by the Lagrangian absolute dispersion, showing that all models correctly reproduce the large scales of coherent eddies. The Lagrangian relative dispersion (or pair dispersion) is more sensitive to structure at smaller scales in the velocity field. The Leith model has the least‐accurate relative dispersion, while the two Jansen and Held models have the most accurate. In initialized forecast experiments, there is very little difference in performance between the models.