We compute annihilation rates of metastable magnetic skyrmions using a form of Langer's theory in the intermediate-to-high damping (IHD) regime. For a Néel skyrmion, a Bloch skyrmion, and an antiskyrmion, we look at two possible paths to annihilation: collapse and escape through a boundary. We also study the effects of a curved vs. a flat boundary, a second skyrmion and a non-magnetic defect. We find that the skyrmion's internal modes play a dominant role in the thermally activated transitions compared to the spin-wave excitations and that the relative contribution of internal modes depends on the nature of the transition process. Our calculations for a small skyrmion stabilized at zero-field show that collapse on a defect is the most probable path. In the absence of a defect, the annihilation is largely dominated by escape mechanisms, even though in this case the activation energy is higher than that of collapse processes. Escape through a flat boundary is found more probable than through a curved boundary. The potential source of stability of metastable skyrmions is therefore found not to lie in high activation energies, nor in the dynamics at the transition state, but comes from entropic narrowing in the saddle point region which leads to lowered attempt frequencies. This narrowing effect is found to be primarily associated with the skyrmion's internal modes.arXiv:1802.06744v7 [cond-mat.mes-hall]