We carry out simulations of gravitationally unstable disks using smoothed particle hydrodynamics(SPH) and the novel Lagrangian meshless finite mass (MFM) scheme in the GIZMO code (Hopkins 2015). Our aim is to understand the cause of the non-convergence of the cooling boundary for fragmentation reported in the literature. We run SPH simulations with two different artificial viscosity implementations, and compare them with MFM, which does not employ any artificial viscosity. With MFM we demonstrate convergence of the critical cooling time scale for fragmentation at β crit ≈ 3.. Non-convergence persists in SPH codes, although it is significantly mitigated with schemes having reduced artificial viscosity such as inviscid SPH (ISPH) (Cullen & Dehnen 2010). We show how the non-convergence problem is caused by artificial fragmentation triggered by excessive dissipation of angular momentum in domains with large velocity derivatives. With increased resolution such domains become more prominent. Vorticity lags behind density due to numerical viscous dissipation in these regions, promoting collapse with longer cooling times. Such effect is shown to be dominant over the competing tendency of artificial viscosity to diminish with increasing resolution. When the initial conditions are first relaxed for several orbits, the flow is more regular, with lower shear and vorticity in non-axisymmetric regions, aiding convergence. Yet MFM is the only method that converges exactly. Our findings are of general interest as numerical dissipation via artificial viscosity or advection errors can also occur in grid-based codes. Indeed for the FARGO code values of β crit significantly higher than our converged estimate have been reported in the literature. Finally, we discuss implications for giant planet formation via disk instability.