Magnetic skyrmions are whirls in the magnetization whose topological winding number promises stability against thermal fluctuations and defects. They can only decay via singular spin configurations. We analyze the corresponding energy barriers of skyrmions in a magnetic monolayer for two distinct stabilization mechanisms, i.e., Dzyaloshinskii-Moriya interaction (DMI) and competing interactions. Based on our numerically calculated collapse paths on an atomic lattice, we derive analytic expressions for the saddle point textures and energy barriers of large skyrmions. The sign of the spin stiffness and the sign of 4th-order derivative terms in the classical field theory determines the nature of the saddle point and thus the height of the energy barrier. In the most common case for DMI-stabilized skyrmions, positive stiffness and negative 4th-order term, the saddle point energy approaches a universal upper limit described by an effective continuum theory. For skyrmions stabilized by frustrating interactions, the stiffness is negative and the energy barrier arises mainly from the core of a singular vortex configuration. arXiv:1907.12174v1 [cond-mat.str-el]